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Abstract 


Early  research  on  time  phasing  primarily  focuses  on  the  theoretical  foundation  for 
applying  the  continuous  distribution  function,  or  S-curve,  to  model  the  distribution  of 
development  expenditures.  Minimal  methodology  is  provided  for  estimating  the  S- 
curve’s  parameter  values.  Brown,  White,  and  Gallagher  (2002)  resolve  this  shortcoming 
through  regression  analysis,  but  their  methodology  has  not  been  widely  adopted  by 
aircraft  cost  analysts,  as  it  is  judged  as  overly  broad  and  not  specific  to  aircraft.  Instead, 
analysts  commonly  apply  the  60/40  “rule  of  thumb”  to  aircraft  development,  assuming  60 
percent  expenditures  at  50  percent  schedule.  It  is  currently  unknown  if  the  60/40 
heuristic  accurately  describes  contemporary  aircraft  development  programs.  Therefore, 
using  a  sample  of  26  DoD  aircraft  programs,  we  first  test  the  accuracy  of  60/40, 
discovering  that,  as  a  heuristic,  the  60/40  cannot  account  for  differences  between  new 
start  and  upgrade  programs.  Next,  we  improve  upon  prior  research  by  using  program 
characteristics  to  construct  an  aircraft-specific  methodology  for  estimating  parameters. 
Finally,  we  conclude  our  research  by  comparing  the  accuracy  of  our  Rayleigh,  Weibull, 
and  Beta  S-curve  models.  Our  Weibull  model  explains  74.6  percent  of  total  variation  in 
annual  budget,  improving  the  estimation  of  budgets  by  6.5  percent,  on  average,  over  the 
baseline  60/40  model. 
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ACCURACY  OF  TIME  PHASING  AIRCRAFT  DEVELOPMENT  USING  THE 


CONTINUOUS  DISTRIBUTION 

I.  Introduction 

Once  a  point  estimate  is  established  for  a  Department  of  Defense  (DoD)  research, 
development,  test  and  evaluation  (RDT&E)  program,  the  program  manager  must  “time 
phase”,  or  spread,  funding  across  the  program’s  expected  fiscal  years.  One  method 
presented  by  current  DoD  cost  analysis  literature  is  the  application  of  the  continuous 
probability  distribution,  commonly  referred  to  as  an  S-curve  (AFCAH,  2007:  15-25).  A 
comprehensive  literature  review  reveals  that  the  single-parameter  Rayleigh  distribution  (a 
special  case  of  the  Weibull  distribution)  is  the  most  frequently  studied  and  applied  S- 
curve  model  for  time  phasing  development  efforts  (Norden,  1970;  Putnam,  1978; 
Watkins,  1982;  Abernathy,  1984;  Lee  et  ah,  1997).  Recently  published  research  shows 
that  increased  precision  may  be  achieved  through  the  application  of  the  two-parameter 
Weibull  and  Beta  distributions  (Brown  et  al.,  2002;  Burgess,  2006).  However,  no 
aircraft  development-specific  means  currently  exists  to  estimate  the  parameters  for  the 
Rayleigh,  Weibull,  and  Beta  distributions;  as  a  result,  many  cost  analysts  instead  favor 
the  application  of  a  60  percent  expenditures  at  50  percent  schedule  “rule  of  thumb”  (Lee 
et  al.,  1997;  NASA,  2002). 

Therefore,  this  research  has  three  goals.  First,  this  research  will  attempt  to 
identify  the  source  of  the  60  percent  expenditures  by  50  percent  schedule  heuristic,  and 
test  its  applicability  against  a  database  of  contemporary  aircraft  development  programs. 
Second,  this  research  will  develop  a  standard  methodology  for  estimating  parameters  for 
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the  Rayleigh,  Weibull,  and  Beta  distributions.  Finally,  using  the  developed  methodology 
for  estimating  parameters,  this  research  will  compare  the  predictive  ability  of  the 
Rayleigh,  Weibull,  and  Beta  distribution  models,  with  the  intent  of  defining  a  best-fit, 
robust  model  to  be  utilized  for  future  time  phased  estimates  of  aircraft  development 
expenditures  and  budgets. 

Problem  Statement 

One  common  method  for  allocating  dollars  over  fiscal  years  is  the  application  of 
the  continuous  distribution,  often  referred  to  as  an  S -curve  within  DoD  cost  estimating 
literature.  In  applying  the  S-curve,  the  Air  Force  Cost  Analysis  Handbook  (AFCAH) 
cautions  that: 

No  single  S-curve...  describes  the  funding  profile  of  all  development  programs; 
rather,  estimators  adjust  the  general  S-shape  of  the  curve  to  model  a  program’s 
particular  expenditure  pattern  of  more/less  effort  earlier/later  in  the  development 
program  (AFCAH,  2007:  15-26). 

However,  the  problem  arises  that  no  standard  baseline  exists  from  which  a  program 
manager  may  begin  analysis.  The  closest  to  a  commonly  accepted  baseline  is  the  60/40 
S-curve,  which  is  cited  as  a  “rule  of  thumb”  by  the  NASA  Cost  Estimating  Guide  (2002: 
168)  and  Lee  et  al.  (1997),  and  used  as  an  example  curve  in  both  the  DoD  Basic  Cost 
Estimating  (BCE)  (2005)  and  AFCAH  (2007)  literature.  Unfortunately,  the  origin  of  the 
60/40  S-curve  remains  unpublished  and  therefore  unclear  to  many  cost  analysts,  making 
it  difficult  to  justify  to  decision  makers.  To  further  convolute  the  rule  of  thumb,  Air 
Force  guidance  reports  that  the  ratio  60/40,  which  describes  the  S-curve’s  skew,  may  be 
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interpreted  to  have  two  different  definitions  (AFCAH,  2007).  Under  the  first,  less 
commonly  used  definition,  60/40  would  describe  a  program  with  60  percent  expenditures 
at  40  percent  schedule.  Under  the  second,  more  commonly  accepted  definition,  60/40 
would  describe  a  program  with  60  percent  expenditures  at  50  percent  schedule.  For  these 
reasons,  the  cost  analyst  may  be  apprehensive  to  accept  the  60/40  S-curve  as  a  standard 
baseline  for  aircraft  development.  And  if  the  60/40  S-curve  is  accepted  as  a  baseline,  no 
published  methodology  exists  for  further  adjusting  the  distribution  parameters  based  on 
program  characteristics.  For  example,  it  is  currently  unknown  whether  the  distribution  of 
expenditures  for  upgrade  programs  differs  from  the  distribution  for  new  starts. 

Therefore,  it  becomes  evident  that  a  requirement  exists  for  an  aircraft-specific  S-curve 
model  capable  of  being  adjusted  by  the  development  program’s  characteristics.  This 
requirement  leads  us  to  develop  our  research  approach,  which  is  presented  next. 

Research  Approach 

The  real  world  applicability  of  past  S-curve  research  has  been  limited,  as  many 
researchers  attempted  a  “one  size  fits  all”  approach  of  defining  an  optimal  Rayleigh 
distribution  for  fitting  all  development  programs.  Limited  consideration  was  given  to 
how  a  program’s  characteristics  drive  the  parameters  of  the  selected  Rayleigh  or  Weibull 
distribution.  Brown,  White,  and  Gallagher  (2002)  resolve  this  limitation,  utilizing  a 
program’s  length  of  development,  program  type,  and  branch  of  service  to  estimate 
Weibull  parameter  values.  Using  their  Weibull  technique,  Brown  et  al.  report  an  average 
correlation  of  0.607  between  the  actual  budget  distribution  and  the  Weibull-predicted 
budget  distribution. 
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As  a  follow-on  effort  to  Brown  et  al.’s  research,  we  will  pursue  a  similar  research 
approach.  As  summarized  in  Figure  1,  this  approach  proposes  that  development  program 
characteristics  may  be  used  to  estimate  distribution  parameters,  which  in  turn  predict  the 
distribution  of  expenditures.  The  distribution  of  expenditures  is  later  converted  to  a 
predicted  budget  profde  before  measuring  goodness-of-fit  (R2).  However,  our  research  is 
uniquely  different  from  Brown  et  ah,  as  we  will  consider  aircraft-specific  development 
characteristics,  such  as  time  to  first  flight  and  aircraft  type,  within  the  model.  We 
postulate  that  the  inclusion  of  these  aircraft-specific  characteristics  will  lead  to  greater 
predictive  ability,  in  turn  leading  to  more  appropriate  time-phased  projections  of  future 
aircraft  development  budgets. 


Aircraft  Program 

Distribution 

Distribution  of 

Characteristics 

Parameters 

Expenditures 

Example:  Example:  Example: 

Upgrade  Aircraft  a  =  2. 1  %  Expenditures  = 

-  0.3  (Upgrade) 

schedule  a 

First  Flight  at  40°o  |3  =  0.44  1  —  e  2 

Schedule  -  0.25  (First  Flight) 

Figure  1:  Research  Approach  for  Aircraft  Development  S-curve  Model 
Research  Implications 

Research  by  Belcher  and  Dukovich  (1999)  documents  that  funding  provided  in 
the  wrong  fiscal  years  of  a  development  program  result  in  productivity  inefficiencies, 
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schedule  slips  and  increased  program  costs.  Unger  (2001)  confirms  these  findings,  and 
coins  the  term  “inappropriate  funding”  to  describe  programs  which  initially  have 
adequate  total  funding,  but  receive  the  funding  in  the  wrong  fiscal  years.  Examples  of 
inappropriate  funding  are  given  by  AFCAH  (2007),  which  warns  some  analysts  resist 
“putting  adequate  monies  in  the  front  end  of  a  new  program  to  fund  technical  personnel 
and  analysts  when  the  program  most  needs  them.  A  program  can  inadvertently  weaken 
itself  by  stretching  this  requirement  out  over  a  number  of  fiscal  years.”  (AFCAH,  2007: 
15-9).  Similarly,  Better  Buying  Power  1.0  (2010)  proposes  that  underfunding  the  peak 
years  of  a  development  program  results  in  a  “leisurely  acquisition  timeline”  and  schedule 
growth.  Inappropriate  funding  leads  to  schedule  growth,  which  consequently  leads  to 
cost  growth. 

As  all  programs  compete  for  funding,  the  usual  result  is  that  a  program  settles  into 
a  level-of-effort  pattern  of  annual  funding  that  does  not  deviate  much  from  year  to 
year. .  .thus  a  one-year  extension  of  a  program  set  to  complete  in  10 
years. .  .result[s]  in  10  percent  growth  in  cost  as  the  team  working  on  the  project  is 
kept  on  another  year  (BBP  F0,  2010:  4). 

Therefore,  it  is  implied  by  Belcher  and  Dukovich  (1999),  Unger  (2001),  AFCAH  (2007) 
and  BBP  1.0  (2010)  that  an  adequate  cost  estimate  may  be  ruined  by  inappropriate  time 
phasing  and  funding  shortfalls,  which  in  turn  create  schedule  and  cost  growth.  To  finish 
on-time  and  on-budget,  a  program  requires  both  adequate  funding  and  appropriate  time 
phasing.  As  a  result,  the  objective  of  this  research  is  to  minimize  schedule  and  cost 
growth  by  allowing  the  cost  analyst  to  more  appropriately  time  phase  his  or  her  aircraft 
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development  program.  To  meet  this  objective,  we  will  next  present  the  investigative 
questions  that  form  the  basis  of  our  research. 

Research  Questions 

The  objective  of  this  research  is  best  summarized  by  the  following  questions: 

1  -  Is  the  “rule  of  thumb”  that  60  percent  of  expenditures  occur  by  50  percent  schedule 
(60/40  S-curve)  accurate  for  contemporary  aircraft  development  programs? 

2  -  What  program  and/or  schedule  characteristics  best  predict  distribution  parameters? 

3  -  Which  distribution  (Rayleigh,  Weibull,  or  Beta)  provides  the  best  S-curve  model  for 
time  phasing  contemporary  aircraft  development  programs? 

Summary 

Current  acquisition  policy  emphasizes  the  importance  of  developing  an  accurate 
point  estimate,  while  undervaluing  the  importance  of  applying  an  appropriate 
methodology  to  the  time  phasing  of  the  estimate.  Research  has  already  established  that 
funding  provided  in  the  wrong  fiscal  year  of  the  program  may  result  in  both  cost  and 
schedule  inefficiencies  (Belcher  et  ah,  1999)  (Unger,  2001).  Therefore,  we  propose  the 
development  of  an  S-curve  model  capable  of  more  accurately  predicting  aircraft 
development  expenditures. 

In  the  next  chapter,  Chapter  2,  the  common  methods  for  time  phasing,  historical 
S-curve  research,  origins  of  the  60/40  rule  of  thumb,  and  attributes  of  each  distribution- 
type  are  discussed.  Chapter  3  reviews  our  data  collection  procedures  and  proposed 
methodology  for  the  estimation  of  a  best-fit  S-curve  model.  Chapter  4  presents  the 
results  of  our  applied  methodology;  in  particular,  the  R“  and  robustness  of  the  model  is 
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reviewed.  Finally,  Chapter  5  summarizes  and  discusses  our  results,  outlines  research 
limitations,  and  suggests  topics  for  future  aircraft  time  phasing  research. 
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II.  Literature  Review 


Chapter  Overview 

This  chapter  is  an  overview  of  topics  related  to  time  phasing  and  the  application 
of  the  S-curve.  First,  the  purpose  and  three  common  methods  for  time  phasing  are 
provided.  Next,  a  literary  review  of  the  evolution  of  the  S-curve  time  phasing  method  is 
presented,  in  chronological  order.  Subsequently,  a  literature  review  identifying  the  origin 
of  the  60/40  “rule  of  thumb”  is  offered.  Finally,  this  chapter  concludes  with  a  brief 
review  of  the  characteristics  of  the  Rayleigh,  Weibull,  and  Beta  distributions.  For  the 
cost  analysis  practitioner  unfamiliar  with  these  distributions,  the  effect  of  increasing  or 
decreasing  the  distribution  parameters  is  shown. 

Methods  of  Time  Phasing 

Contemporary  cost  analysis  literature  offers  three  methods  for  time  phasing: 
schedule/milestone,  analogy,  and  the  S-curve  (AFCAH,  2007).  The  choice  of  one 
method  over  another  method  is  often  influenced  by  the  availability  of  schedule 
information  on  the  program  to  be  estimated,  as  well  as  the  availability  of  historical  data 
on  analogous  programs. 

The  schedule/milestone  method  is  considered  the  most  exact  means  of  phasing, 
but  it  is  also  the  most  complex.  When  completed  correctly,  the  schedule/milestone 
method  is  easily  explained  to  decision  makers,  and  is  highly  defendable  during  outside 
analyses.  However,  the  accurate  application  of  the  schedule/milestone  method  requires  a 
master  program  schedule  and  work  breakdown  structure  (WBS)  that  may  be  unattainable 
for  the  initial  time  phased  estimate  of  a  RDT&E  program  (AFCAH,  2007:  15-15).  When 
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a  master  schedule  is  not  available,  the  cost  analyst  must  either  estimate  the  schedule  or 
rely  on  an  alternate  time  phasing  method  (BCE,  2005:  16-6). 

By  comparison,  the  analogy  method  is  the  simplest  method  as  it  utilizes  the  time 
phased  expenditures  of  a  single,  analogous  development  program  as  the  basis  for 
spreading  funding.  However,  AFCAH  instructions  warn  “the  process  of  finding  a  truly 
analogous  program.,  .may  be  difficult  and  time  consuming”  (AFCAH,  2007:  15-21). 
Additionally,  the  analogy  method’s  reliance  on  a  single  program  may  introduce 
additional  uncertainty  and  risk  which  could  be  mitigated  through  the  use  of  the  S-curve, 
which  accrues  data  from  multiple  programs. 

Therefore,  the  use  of  the  S-curve,  or  cumulative  distribution  function  (CDF),  is 
often  the  preferred  method  for  RDT&E  programs  initiated  with  considerable  schedule 
uncertainty.  When  contrasted  with  the  analogy  method,  the  S-curve  is  theoretically 
superior,  as  it  may  utilize  historical  data  from  multiple  analogous  programs  to  form  an 
estimated  distribution  of  expenditures.  The  resulting  expenditure  distribution  may  then 
be  adjusted  based  on  unique  program  characteristics  expected  to  shift  effort  earlier  or 
later  into  the  program’s  development  (AFCAH,  2007:  15-25).  As  shown  in  Figure  2,  the 
S-curve  is  given  its  name  from  the  shape  formed  when  cumulative  program  expenditures 
are  plotted  over  time. 
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Figure  2:  S-Curve  of  Cumulative  Costs  over  Time 


Evolution  of  the  S-curve  in  Cost  Analysis  Literature 

No  single  continuous  distribution  type  describes  the  S-curve.  As  shown  in  Table 
1,  a  review  of  literature  reveals  that  a  variety  of  continuous  distributions  may  be  used  to 
model  the  relationship  between  a  program’s  cost  and  time.  Beginning  with  Norden’s 
publication  of  “Useful  Tools  for  Project  Management”  in  1970,  the  single  parameter 
Rayleigh  distribution  is  the  most  frequently  applied  and  researched  application  of  the  S- 
curve.  Conversely,  Weida  (1977)  offers  the  solution  of  applying  two  quadratic  equations. 
Brown  et  al.  and  Unger  et  al.  subsequently  improve  upon  previous  research,  showing  that 
the  two  parameter  Weibull  distribution  offers  a  more  robust  fit  for  RDT&E  programs 
(Brown  et  al.,  2002;  Unger  et  al.,  2004).  Finally,  Burgess  (2006)  simultaneously  tests  the 
Rayleigh,  Weibull,  and  Beta  distributions;  his  results  show  that  the  Weibull  distribution 
outperforms  both  the  Rayleigh  and  Beta  distributions  in  fitting  expenditure  patterns  for 
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26  DoD  RDT&E  space  contracts  (Burgess,  2006:  24-25).  For  the  remainder  of  this 
subchapter,  we  will  review  the  findings  of  these  publications  in  greater  detail. 


Table  1:  Notable  S-curve  Research 


Year 

Author 

Distribution  Applied 

1970 

Norden 

Rayleigh 

1977 

Weida 

Quadratic  Equation 

1978 

Putnam 

Rayleigh 

1982 

Watkins 

Rayleigh 

1984 

Abernethy 

Rayleigh 

1997 

Lee  et  al. 

Rayleigh 

2001 

Unger 

Rayleigh  and  Weibull 

2002 

Brown  et  al. 

Rayleigh  and  Weibull 

2006 

Burgess 

Rayleigh,  Weibull,  and  Beta 

Applications  of  the  Rayleigh  Distribution  (Special  Case  of  the  Weibull) 

Norden  (1970)  first  applies  the  continuous  distribution  to  development 
expenditures  by  utilizing  the  Rayleigh  distribution,  which  is  a  special  case  of  the  Weibull 
Distribution,  to  relate  monthly  manpower  usage  and  elapsed  project  time.  Norden 
observes  that  “there  are  regular  patterns  of  manpower  buildup  and  phase-out  in  complex 
project.  The  cycles. .  .seem  to  be  a  function  of  the  way  groups  of  engineers  and  scientists 
tackle  complex  technological  development  problems”  (Norden,  1970:  80).  By  using  the 
assumption  that  the  rate  of  problem  solving  increases  as  a  linear  function  of  time,  a 
relationship  may  be  derived  between  manpower  and  project  time.  Nordon  represents  this 
relationship  using  the  Rayleigh  cumulative  distribution  function 

y  =  K(l  —  e~at2)  (1) 

where  y  =  cumulative  manpower  utilized  at  time  t,  K  =  total  cumulative  manpower 
utilized  by  end  of  the  project,  a  =  Rayleigh  scale  parameter,  and  t  =  elapsed  time  since 
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the  start  of  the  project.  Norden  states  that  on  average,  projects  of  high  importance  will 
reach  peak  manpower  sooner,  while  projects  of  lower  importance  will  take  longer  to 
reach  peak  effort.  Therefore,  highly  important  projects  are  numerically  represented  by  a 
larger  scale  parameter  a;  projects  of  lower  importance  are  represented  with  a  smaller 
scale  parameter  a. 

Putnam  (1978)  builds  on  Norden’s  approach  of  applying  the  Rayleigh  distribution 
to  RDT&E  projects.  Putnam  relates  Norden’s  Rayleigh  model  to  software  development 
by  analyzing  the  recorded  manpower-hours  of  50  U.S.  Anny  Computer  Systems 
Command  contracts.  Putnam  calculates  that  the  peak  (inflection  point)  of  manpower 
effort  occurs  at  39.45  percent  of  software’s  development  lifecycle  (Putnam,  1978:  349). 
These  findings  validate  Norden’s  theory  that,  on  average,  manpower  requirements  for 
development  projects  have  a  shorter  rise  and  longer  exponential  tail.  However,  Putnam 
also  notes  that  a  small  subset  of  contracts  within  his  dataset  do  not  follow  the  expected 
Rayleigh  distribution,  and  instead  exhibit  a  stepwise  increase  to  peak  effort,  followed  by 
steady  effort  requirements  until  completion.  Putnam  postulates  that  projects  which  do 
not  confonn  to  the  Rayleigh  distribution  are  the  result  of  inefficient  management  acting 
contrary  to  system  requirements.  Putnam  summarizes  his  assessment  by  saying  that 
“usually  management  adapts  to  the  system  signals,  but  generally  responds  late  because 
the  signal  is  not  clear  instantaneous  with  the  need”  (Putnam,  1978:  348).  In  such  a  case, 
the  project  is  overfunded  or  overstaffed  during  the  later  years  or  months  of  the 
development  program,  resulting  in  management  continuing  to  support  peak  manpower 
levels  after  the  manpower  is  no  longer  required. 
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Following  Putnam,  cost  researchers  Watkins  and  Abernethy  further  confirm  the 
viability  of  the  Rayleigh  distribution  for  time  phasing.  Watkins  (1982)  uses  the  Rayleigh 
distribution  to  forecast  cost  growth  in  contracts.  By  applying  the  shape  of  the  Rayleigh 
distribution  against  the  difference  between  actual  cost  of  work  performed,  budgeted  cost 
of  work  performed,  and  budget  cost  of  work  scheduled,  Watkins  hypothesizes  that  total 
cost  growth  (as  defined  by  budget  at  completion)  may  be  estimated  early  within  a 
development  contract.  Abernethy  (1984)  models  the  Rayleigh  distribution  against  cost 
data  from  21  Navy  contracts,  to  include  Navy  aircraft  and  missiles.  Although  Abernethy 
successfully  fits  the  Rayleigh  distribution  to  individual  contracts,  he  cannot  produce  a 
Rayleigh  parameter  that  fits  all  contract  data  to  within  five  percent  of  its  final  values. 
Abernethy  theorizes  that  an  increased  sample  size  will  result  in  additional  precision  for 
future  studies. 

Finally,  cost  researchers  Lee  et  al.  (1997)  offer  two  theory-based  contributions  to 
the  Rayleigh  S-curve.  First,  Lee  et  al.  write  that  it  has  been  previously  observed  that  the 
peak  expenditure  rate  (tp)  for  “aircraft  development  programs  often  comes  at,  or  slightly 
before,  the  time  of  first  flight”  (Lee  et  al,  1997:32).  Therefore,  the  Rayleigh’s  scale 
parameter  a  may  be  calculated  by  setting  the  derivative  of  the  Rayleigh’s  probability 
distribution  function  equal  to  0  at  the  time  of  first  flight.  When  simplified,  this  equates  to 
a  —  0.5 tp  for  the  Rayleigh  equation  given  in  (1).  Second,  Lee  et  al.  offer  a  solution 
for  those  applying  the  60/40  S-curve.  By  setting  a  —  3.5,  the  Rayleigh  distribution  will 
always  produce  a  S-curve  for  which  60  percent  expenditures  occur  at  50  percent 
schedule,  when  truncated  to  control  for  the  Rayleigh’s  infinite  tail.  Lee  et  al.  truncate 
their  S-curve  model  by  assuming  that  schedule  completion  occurs  at  97  percent  of 
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expenditures.  It  should  be  noted  that  we  will  again  reference  Lee  et  al.’s  60/40  Rayleigh 
S-curve  in  Chapter  4  of  this  thesis,  as  the  model  will  serve  as  our  standard  “baseline” 
model  for  comparison. 

Application  of  the  Quadratic  Function 

Rather  than  rely  on  an  established  distribution,  Weida  (1977)  offers  the  approach 
of  using  quadratic  functions  to  estimate  the  S-curve.  Weida’s  method  estimates  two 
independent  quadratic  equations;  one  quadratic  equation  represents  all  expenditures 
before  the  inflection  point,  with  the  other  equation  representing  expenditures  following 
inflection.  After  aggregating  the  cost  performance  reports  from  17  DoD  weapon  systems, 
Weida  finds  that  the  mean  inflection  point  for  aircraft  development  contracts  occurs  at 
56.2  percent  of  expenditures  and  45.2  percent  of  time  (Weida  1977:  8). 

Applications  of  the  Weibull  &  Beta  Distributions 

Unger  (2001)  first  recommends  that  the  Weibull  distribution  is  a  better  predictor 
of  RDT&E  expenditure  profiles  than  the  Rayleigh  distribution.  Unger  tests  the  ability  of 
both  the  Rayleigh  and  Weibull  to  predict  variation  and  cost  and  schedule  growth,  finding 
that  the  Weibull  outperforms  the  Rayleigh  when  fit  to  individual  programs.  However,  in 
his  findings,  Unger  annotates  a  significant  limitation  of  his  model:  no  method  currently 
exists  to  estimate  the  Rayleigh  and  Weibull  parameters  for  future  programs. 

Brown,  White,  and  Gallagher  (2002)  resolve  Unger’s  stated  limitation,  using 
multi-stage  regression  techniques  to  estimate  Rayleigh  and  Weibull  parameters.  Brown 
et  al.  define  their  Weibull  cumulative  distribution  function  as 

F(t)  =  1  -  (2) 
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where  t  =  time,  y  =  location  parameter,  8  =  scale  parameter,  and  /?  =  shape  parameter. 
Using  128  completed  DoD  RDT&E  programs,  Brown  et  al.  estimate  that  the  Weibull 
scale  parameter  8  is  a  linear  function  of  program  duration,  defined  as 
8  —  0.726(Duration).  Similarly,  the  shape  parameter  /?  is  a  function  of  duration, 
branch  of  service,  and  program  type,  and  defined  as  =  1.299  +  0.972  In  (Duration) 

—  0A61(NonAF)  —  0 .543(Electronic)  —  1.099  (Space).  Brown  et  al.  report  their 
scale  parameter  model  coefficient  of  determination  as  0.922;  the  shape  parameter  model 
coefficient  of  determination  is  0.309. 

In  addition  to  duration,  branch  of  service,  and  program  type,  Brown  et  al.  test  the 
predictive  ability  of  total  program  cost,  but  find  that  it  does  not  have  a  statistically 
significant  influence  on  the  Weibull’s  scale  and  shape  parameters.  Furthermore,  they 
find  that  the  Weibull  parameter  estimates  for  missiles,  munitions,  ships,  and  vehicles  are 
statistically  equivalent  to  aircraft.  When  contrasting  their  Weibull  model  with  their 
Rayleigh  model,  Brown  et  al.  detennine  the  Weibull  distribution  improves  budget  profile 
projection  by  60  percent,  on  average.  Brown  et  al.  conclude  their  research  by  observing 
that  because  the  Rayleigh  distribution  is  limited  to  one  scale  parameter,  the  Rayleigh 
distribution  often  overestimates  expenditures  during  the  early  phase  of  programs  and 
underestimates  during  the  later  phases  (Brown  et  al.,  2002:  5 1). 

Following  Brown  et  al.,  Burgess  (2006)  further  advances  S-curve  estimation 
techniques  by  simultaneously  testing  the  Rayleigh,  Weibull,  and  Beta  distributions.  Using 
contract  expenditure  data  from  26  National  Reconnaissance  Office  and  DoD  space 
satellite  systems,  Burgess  concludes  that  although  the  Beta  distribution  is  more  accurate 
than  the  Rayleigh,  it  is  less  accurate  than  the  Weibull.  Burgess  reports  that  it  is  not 
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surprising  that  the  Weibull  and  Beta  distribution  show  greater  accuracy  than  the 
Rayleigh,  “because  the  Weibull  and  Beta  have  more  parameters  and  are  therefore  better 
able  to  accommodate  variations  among  spending  profiles”  (Burgess,  2006:  17). 

However,  Burgess  is  not  satisfied  with  the  ability  of  the  Weibull  distribution  to  fit  the 
aggregated  data  set  used  within  his  study. 

In  particular,  Burgess  observes  that  the  Weibull  model  fails  to  capture  late 
program  spending.  This  shortfall  in  predictive  capability  is  the  result  of  a  theoretical 
assumption  of  the  distribution;  when  applying  a  continuous  distribution,  we  must  assume 
that  the  rate  of  work  (and  thus  rate  of  spending)  is  completely  variable  and  always  begins 
and  ends  at  zero.  This  assumption  is  not  consistent  with  reality,  where  a  fixed  cost  exists 
for  each  year  of  a  space  program.  Therefore,  Burgess  recommends  the  inclusion  of  a 
constant  rate  term  in  the  Weibull  equation,  based  on  schedule  length,  to  better  account  for 
the  fixed  costs  and  overhead  present  within  satellite  RDT&E  programs  (Burgess  2006: 
25).  Burgess  publishes  his  modified  Weibull  equation  as 

E(t)  =  d[Rt  +  1  -  e~atP]  (3) 

where  d  —  c°^a,  a  —  —0.414  +  0.729 (units)  +  0.0488 (months  duration )  + 

0. 145 (%  nonrecurring),  /?  =  1.71,  and  R  =  0.0 0148 (months  duration ). 

Origin  and  Applicability  of  the  60/40  “Rule  of  Thumb” 

Introduced  in  Chapter  1,  the  60/40  “rule  of  thumb”  is  a  commonly  cited  heuristic 
assumption  used  to  approximate  the  continuous  distribution.  It  is  still  in  use  by  cost 
analysts  to  time  phase  aviation  programs,  as  confirmed  by  AFLCMC-FZC  staff. 

Notably,  the  use  of  the  60/40  S-curve  is  not  specific  to  aircraft  cost  analysis,  as  a 
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contemporary  review  of  cost  analysis  literature  reveals  that  the  NASA  Cost  Estimating 
Guide  (2002)  references  the  60/40  S-curve  as  a  “rule  of  thumb”  for  spreading  space 
expenditures  (but  does  not  provide  an  origin  or  reference).  Additionally,  the  DoD’s  BCE 
(2005)  and  Air  Force’s  AFCAH  (2007)  repeatedly  use  the  60/40  S-curve  as  an  example 
within  time  phasing  chapters;  however,  both  documents  fail  to  provide  a  source. 
Therefore,  we  search  for  the  origin  of  the  60/40  S-curve  to  better  judge  its  applicability 
for  time  phasing  contemporary  aircraft  development. 

We  discover  that  Lee  et  al.  (1997)  trace  the  most  likely  origin  of  the  rule  of  thumb 
to  a  circa- 1980  aircraft  study  completed  by  the  OSD  Cost  Analysis  Improvement  Group 
(CAIG).  The  CAIG  study  remains  undocumented,  but  the  results  are  recalled  by  Mr. 
Gary  Christie,  former  Deputy  Director  of  Acquisition  Management  at  OSD  (Lee  et  al., 
1997:33).  Since  the  CAIG  study  remains  unpublished  and  is  no  longer  recoverable,  it  is 
not  possible  to  determine  the  applicability  of  the  study’s  methodology  and  data  source. 
Therefore,  we  instead  evaluate  the  accuracy  of  the  60/40  rule  of  thumb  by  providing  a 
historical  synopsis  of  findings  from  all  available  aircraft  development  time  phasing 
studies: 

1  -  Weida  (1977)  reviews  seventeen  aircraft  development  contracts  and  detennines  that 
contracts  obligate  57  percent  of  expenditures  by  50  percent  schedule,  on  average. 
Additionally,  Weida  provides  a  table  of  “inflection  points”,  which  identify  the  time  of 
peak  expenditures  within  a  development  program.  Weida’s  inflection  point  findings  are 
reproduced  in  Appendix  1 . 
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2  -  In  1979,  a  study  completed  by  the  General  Research  Corporation  finds  that  aircraft 
contracts  typically  obligate  57  percent  of  expenditures  by  50  percent  schedule.  An 
extensive  literary  search  could  not  locate  the  original  General  Research  Corporation 
study;  instead,  we  rely  on  a  summation  of  the  General  Research  Corporation  study 
provided  by  Dibbly  (1988:  14). 


3  -  Dibbly  (1988)  reviews  22  aircraft  avionics  development  contracts  and  discovers  that 
contract  type,  contract  value  and  engineering  complexity  have  a  statistically  significant 
effect  on  the  S-curve  distribution.  Based  on  the  combination  of  these  characteristics, 
aircraft  avionics  development  contracts  will  fall  into  one  of  three  categories.  As 
summarized  in  Table  2,  contracts  will  typically  obligate  either  57  percent  expenditure  at 
50  percent  schedule,  68  percent  expenditures  at  50  percent  schedule,  or  78  percent 
expenditures  at  50  percent  schedule. 

Table  2:  Dibbly  (1988)  Avionics  Funding  Profile  Study 


Contract  Type 

Contract  Value 

Engineering  Complexity 

S-curve 

Cost  Plus 

Small 

High 

Cost  Plus 

Large 

High 

57/43 

Fixed  Price 

Large 

High 

Cost  Plus 

Large 

Low 

Fixed  Price 

Small 

High 

Fixed  Price 

Large 

Low 

68/32 

Cost  Plus 

Small 

Low 

Fixed  Price 

Small 

Low 

78/22 

4  -  Brown  et  al.  (2002)  include  19  aircraft  development  programs  as  part  of  their  larger, 
multi-platform  study  of  the  Weibull-based  S-curve.  Although  Brown  et  al.  do  not 
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provide  a  summary  of  percent  expenditures  by  50  percent  schedule,  we  may  approximate 
this  statistic  using  the  estimated  Weibull  parameters  provided  within  Brown’s  thesis 
appendix  (Brown,  2001:  65-69).  Using  this  data,  we  estimate  that  57.1  percent  of 
expenditures  occur  at  50  percent  schedule,  on  average. 

Characteristics  of  Probability  Distributions 

As  denoted  earlier  in  this  chapter,  the  S-curve  is  given  its  name  from  the  shape 
fonned  when  a  cumulative  density  function  (CDF)  is  fitted  to  the  plot  of  cumulative 
effort  versus  development  time.  The  AFCAH  (2006,  15-25)  reports  that  the  nonnal 
distribution  is  typically  used.  However,  a  significant  limitation  of  the  nonnal  distribution 
is  the  assumption  that  values  are  symmetric  about  the  mean,  or  schedule  midpoint. 
Therefore,  when  applied  to  development  efforts,  the  nonnal  distribution  could  only 
model  programs  which  are  expected  to  expend  equal  funding  during  the  earlier  and  later 
half  of  the  development  schedule,  as  shown  in  Figure  3.  The  normal  distribution’s 
assumption  of  symmetry  is  incongruent  with  published  S-curve  research,  which  find  that, 
on  average,  aircraft  development  programs  expend  a  greater  percentage  of  funds  during 
the  first  50  percent  of  schedule  (Weida,  1977;  Dibbly,  1988;  Lee  et  ah,  1997;  Brown  et 
al„  2002). 

Therefore,  to  provide  a  better  fit  for  the  positively  skewed  expenditure  pattern, 
multi-parameter  distributions  with  one  or  more  flexible  shape  parameters  are  considered. 
Based  on  precedence  established  by  Brown  et  al.  (2002)  and  Burgess  (2006),  the  Weibull 
distribution  (for  which  the  Rayleigh  distribution  is  a  special  case)  and  the  Beta 
distribution  are  selected  for  testing.  A  summary  of  the  Weibull  and  Beta  distribution 
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characteristics  follow.  To  allow  for  easier  comprehension  and  application  of  these 
distributions  by  the  cost  analyst  practitioner,  we  will  utilize  Microsoft  Excel’s  naming 
convention  for  distribution  parameters  for  the  remainder  of  this  thesis,  rather  than  apply 
conventions  more  commonly  utilized  within  mathematical  texts. 


Figure  3:  The  Normal  Distribution 
Characteristics  of  the  Weibull  Distribution 

The  two-parameter  Weibull  distribution  is  named  for  the  Swedish  scientist 
Waloddi  Weibull,  who  in  1951  used  the  distribution  to  model  the  breaking  strength  of 
materials  (Johnson  et  ah,  1994:  628).  Mathematically,  the  Weibull  CDF  consists  of  a 
shape  parameter,  a,  and  scale  parameter,  P  and  is  written  as 

,%  schedule. a 

%  Expenditures  —  1  —  e  P  (4) 

Within  Excel,  this  same  function  is  represented  as 

=  Weibull(%  schedule,  a,  /?,  true )  (5) 
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The  effect  of  modifying  the  Weibull  distribution’s  shape  parameter,  a,  is  shown  in  Figure 
4.  When  a  =  3.7  and  (B  =  0.6,  the  distribution  is  approximately  symmetric,  or  equivalent 
to  the  nonnal  distribution,  and  expenditures  are  equivalent  for  the  first  half  and  last  half 
of  development.  As  we  decrease  the  value  of  the  shape  parameter  a  below  3.7,  while 
holding  the  scale  parameter  (3  constant,  program  spending  increases  during  the  first  half 
of  development  and  the  distribution  becomes  positively  skewed. 


Figure  4:  Effects  of  changing  Weibull  shape  parameter  a 

Conversely,  the  effect  of  modifying  the  Weibull  distribution’s  scale  parameter,  (3, 
which  controls  statistical  dispersion,  is  shown  in  Figure  5.  Again  we  start  with  a  Weibull 
distribution  where  a  =  3.7  and  (3  =  0.6,  which  is  roughly  equivalent  to  the  nonnal 
distribution.  As  scale  parameter  (3  is  decreased,  the  distribution  becomes  more  highly 
peaked  and  spending  varies  more  significantly  from  period  to  period.  Therefore,  a 
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decrease  in  the  value  of  (3,  while  holding  a  constant,  will  result  a  less  unifonn,  more 
“pointed”  distribution  of  expenditures. 


Figure  5:  Effects  of  changing  Weibull  scale  parameter  P 


Characteristics  of  the  Rayleigh  Distribution 

The  one-parameter  Rayleigh  is  a  special  case  of  the  Weibull  distribution  which 
assumes  the  shape  parameter,  a,  is  fixed  at  a  value  of  2,  and  only  the  scale  parameter  P  is 
allowed  to  vary.  As  documented  in  Chapter  1,  the  Rayleigh  fonns  the  basis  of  early  S- 
curve  research  (Norden,  1970;  Putnam,  1978;  Watkins,  1982;  Abernathy,  1984).  The 
Rayleigh’s  applicability  to  early  research  may  be  attributed  to  the  characteristic  that 
Rayleigh-modeled  peak  expenditures  occur  at  38  percent  of  program  schedule,  when 
program  schedule  is  truncated  at  97  percent  of  expenditures  (to  control  for  an  infinite  tail) 
(Lee  et  ah,  1997).  This  characteristic  is  notable,  as  it  results  in  a  distribution  in  which 
approximately  60  percent  of  expenditures  occur  by  50  percent  schedule,  consistent  with 
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the  60/40  rule  of  thumb  introduced  earlier  (Lee  et  ah,  1997)  (Burgess,  2006).  Therefore, 
in  all  cases  the  Rayleigh  distribution  will  be  positively  skewed,  meaning  that  peak 
expenditures  will  occur  during  the  first  half  of  development. 

Characteristics  of  the  Beta  Distribution 

Compared  to  the  Weibull  distribution,  which  consists  of  only  one  higher-order 
shape  parameter  and  one  scale  parameter,  the  Beta  distribution  contains  two  higher-order 
shape  parameters,  a  and  (3.  Therefore,  the  Beta  is  more  versatile,  and  the  distribution’s 
“flexibility  encourages  its  empirical  use  in  a  wide  range  of  applications”  (Johnson  et  ah, 
1995:  235).  In  CDF  form,  the  Beta  equation  is  written  as 

n/  j-,  ..  (l-%  schedule)P~1(%  schedule)a~1r  (a+G) 

%  Expenditures  =  - -  r(«)r(p) - - - 1 — —  (6) 

where  T  represents  a  factorial  function  for  which  F(n)  =  (n  —  1)!  Within  Excel,  this 
equation  is  represented  by  the  command 

=  Betadist(%  schedule,  a,  /?)  (7) 

Although  the  relationship  between  the  a  and  (3  parameters  is  mathematically  very 
complex,  two  broad  observations  may  be  made  relating  to  the  Beta  distribution’s  skew 
and  kurtosis. 

First,  it  is  shown  in  Figure  6  that  the  skewness  of  the  Beta  distribution  is 
influenced  by  the  ratio  between  shape  parameters  a  and  (3.  When  a  =  (3,  the  distribution 
is  symmetric  and  closely  resembles  a  normal  distribution,  and  expenditures  are  equivalent 
for  the  first  half  and  last  half  of  development.  When  a  >  (3,  the  distribution  is  negatively 
skewed,  resulting  in  peak  spending  during  the  last  half  of  development.  When  a  <  (3,  the 
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distribution  is  positively  skewed,  resulting  in  peak  spending  during  the  first  half  of 
development.  Therefore,  it  is  expected  that  a  <  (3  for  most  development  programs. 

Second,  it  is  shown  in  Figure  7  that  as  the  a  and  (3  parameters  increase  in  value 
from  zero,  dispersion  decreases  (and  kurtosis,  or  peakedness,  therefore  increases).  When 
a  =  1  and  (3=1,  the  distribution  is  uniformly  distributed,  which  assumes  that  expenditures 
are  constant  throughout  development.  By  comparison,  when  a  and  P  both  increase  in 
value  to  3,  dispersion  decreases  and  the  “S-curve  shape”  becomes  visible. 


Figure  6:  Effects  of  changing  the  ratio  between  Beta  shape  parameters 
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Figure  7:  Effects  of  changing  the  magnitude  of  Beta  shape  parameters 
Summary 

Funding  provided  in  the  wrong  fiscal  year  of  a  program  may  result  in  lost 
funding,  schedule  slips,  and  cost  growth  (Unger,  2001).  As  a  result,  the  AFCAH,  the 
primary  guidebook  for  Air  Force  cost  analysts,  provides  three  methods  to  increase  time 
phasing  precision:  schedule/milestone,  analogy,  and  S-curve.  For  RDT&E  applications, 
the  S-curve  is  often  the  preferred  method,  as  the  S-curve  does  not  require  detailed 
program  schedule  or  WBS  data  (AFCAH,  2007). 

Initially,  analysts  applying  the  S-curve  time  phasing  method  primarily  relied  on 
the  single  parameter  Rayleigh  distribution.  The  Rayleigh  distribution  is  a  special  case  of 
the  Weibull  distribution  which  assumes  a  fixed  shape  parameter  of  2.  More  recently,  the 
Weibull  and  Beta  distributions  have  become  commonplace  as  Brown  et  al.  (2002),  Unger 
et  al.  (2004),  and  Burgess  (2006)  have  demonstrated  the  increased  precision  available 
from  utilizing  multi-parameter  distributions.  Having  defined  the  Weibull  and  Beta 
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distributions  and  reviewed  previous  research,  we  will  next  define  our  methodology  for 
constructing  an  aircraft-development  specific  S-curve  model. 
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III.  Methodology 


Chapter  Overview 

The  purpose  of  this  chapter  is  to  define  the  methods  used  to  gather  and  analyze 
data  while  constructing  our  model.  First,  we  explain  the  source  for  aircraft  RDT&E  data 
and  our  decision  to  utilize  total  obligation  authority  (TO A)  data  instead  of  contract 
expenditures.  Following  data  source  selection,  our  method  for  converting  TOA  into 
estimated  base  year  2014  expenditures  is  explained.  Next,  we  present  our  method  for 
standardizing  each  expenditures  data  point  into  a  percent  schedule  by  percent 
expenditures  data  base.  Finally,  the  procedure  for  estimating  best-fit  Rayleigh,  Weibull, 
and  Beta  parameters  is  given,  and  the  methodology  for  detennining  the  significance  of 
our  resulting  Rayleigh,  Weibull,  and  Beta  models  is  explained. 

Data  Source 

The  first  step  in  any  research  methodology  is  detennining  the  most  credible  and 
applicable  source  for  data.  Therefore,  the  efforts  of  previous  S-curve  researchers  are 
reviewed.  For  their  research,  Brown  et  al.  (2002),  Porter  and  Gallagher  (2004),  and 
Unger  et  al.  (2004)  rely  on  the  Select  Acquisitions  Report  (SAR)  to  obtain  the  Total 
Obligation  Authority  (TOA),  which  is  the  amount  of  budget  for  each  year  of  a 
development  program.  Using  the  OSD  Comptroller  outlay  rates,  the  authors  are  able  to 
use  TOA  to  approximate  annual  expenditures.  By  comparison,  Norden  (1970),  Weida 
(1977),  Putnam  (1978),  Abernathy  (1984),  and  Burgess  (2006)  obtain  expenditure  data 
directly  from  contractor  cost  reports;  therefore,  no  further  transfonnation  of  data  is 
required  to  obtain  actual  expenditures.  Although  either  method  of  data  sourcing  is 
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acceptable,  we  determine  that  utilizing  the  SAR’s  TOA  data  are  more  desirable  for  two 


reasons. 

First,  we  determine  that  SAR  TOA  data  are  more  applicable  to  the  “real-world” 
time  phasing  estimates  that  occur  early  within  an  aircraft  program’s  development.  Once 
transformed  by  outlay  rates,  TOA  data  directly  estimate  a  development  program’s  total 
expenditures  across  all  development  contracts.  By  comparison,  contract  expenditure  data 
from  a  single  contractor’s  Cost  Data  Summary  Reports  may  only  represent  a  portion  of 
the  total  aircraft  development  effort.  Furthennore,  during  the  research  process,  it  was 
discovered  that  the  prime  contractor  and  supporting  contractors  often  submit  their 
reoccurring  annual  expenditure  reports  in  different  months  of  the  year.  As  a  result,  for 
programs  with  multiple  development  contracts,  it  is  often  not  possible  to  sum 
expenditures  across  individual  contracts  to  obtain  total  program  expenditures.  Due  to  this 
limitation,  we  therefore  judge  SAR  data  as  more  representative  of  the  entire  aircraft 
development  effort. 

Second,  SAR  data  are  more  consistently  reported  and  available.  By  law,  SAR 
reports  are  updated  and  reported  annually  and  include  budgeted  TOA  amounts  for  every 
fiscal  year  of  development.  By  comparison,  a  review  of  Cost  Data  Summary  Reports 
within  DACIMS  revealed  one  or  more  missing  annual  reports  for  19  of  the  26  programs 
in  our  database.  Furthermore,  8  of  26  programs  considered  have  no  Cost  Data  Summary 
Reports  available  for  their  entire  development  effort.  Therefore,  we  determine  that 
utilizing  SAR  data  will  result  in  a  more  robust  database  with  greater  explanatory  power. 
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Data  Selection 


For  the  purposes  of  defining  an  aircraft-specific  S-curve  model  and  testing  the 
60/40  heuristic,  only  aircraft  development  programs  are  incorporated.  Aircraft 
development  programs  are  defined  as  any  fixed-wing,  manned  aircraft  developed  for  one 
or  more  of  the  U.S.  DoD  service  branches  and  designated  as  DoD  Acquisition  Category  1 
(ACAT  1).  After  applying  these  criteria,  we  arrive  at  an  initial  sample  size  of  28 
programs  with  available  SAR  TOA  data.  Next,  we  further  specify  that  programs  must 
have  a  reported  engineering  and  manufacturing  development  (EMD)  contract  award  date, 
first  flight  date,  and  initial  operational  capability  (IOC)  date.  After  applying  these 
criteria,  2  of  the  28  aircraft  development  programs  are  deselected,  leaving  a  remaining 
sample  size  of  26  aircraft  as  shown  in  Table  3.  The  C-130  AMP  and  KC-130J  programs 
are  both  deselected  for  lacking  a  published  first  flight  and  IOC  date. 


Table  3:  Selected  Aircraft  Development  Programs 


1 

A-10 

14 

EA-6B  ICAP 

2 

AWACS  BLOCK  40 

15 

F-14A 

3 

AWACS  RSIP 

16 

F-14D 

4 

B-l  CMUP 

17 

F-15A 

5 

B-2  EHF  1 

18 

F-16A/B 

6 

B-2  RMP 

19 

F-18E/F 

7 

B-2  A 

20 

F-18A 

8 

C-17A 

21 

F-22 

9 

C-5  AMP 

22 

F-35  AF 

10 

C-5  RERP 

23 

F-5E 

11 

E-2D 

24 

J  STARS 

12 

E-6A 

25 

P-8A 

13 

EA-18G 

26 

T-46A 
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Converting  TOA  (Budget)  to  Annual  Expenditures  (Base  Year) 


RDT&E  funding  (appropriation  3600)  is  budgeted  to  execute  in  the  first  fiscal 
year  of  availability,  but  may  be  approved  by  the  acquisition  program’s  Chief  Financial 
Officer  to  obligate  in  the  second  year  of  availability.  Thus,  RDT&E  funding  is 
considered  “multiyear”,  and  available  for  incurring  obligation  for  more  than  one  fiscal 
year  (AFCAH,  2007:  15-4).  Therefore,  when  estimating  annual  expenditures  from  TOA, 
we  must  account  for  this  uncertainty  by  applying  the  OSD-Comptroller  outlay  rate  to 
approximate  multiyear  spending.  As  an  example,  consider  the  published  RDT&E  Navy 
outlay  rates  for  fiscal  years  1992  through  2002  are  shown  in  Table  4.  From  this  table,  we 
may  interpret  that,  on  average,  a  Navy  RDT&E  program  will  spend  55.95  percent  of 
available  FY92  funding  during  FY92  (first  year),  33.01  percent  of  FY92  funding  during 
FY93  (second  year),  and  only  7.97  percent  of  FY92  funding  during  FY94  (third  year). 


Table  4:  Navy  RDTE  Outlay  Rates  (FY92-02) 


Outlay  Rates  (Percentage) 

First  Y ear 

Second  Year 

Third  Y ear 

Fourth  Year 

Fifth  Year 

FY92 

55.95 

33.01 

7.97 

1.43 

1.64 

FY93 

54.36 

33.88 

7.91 

1.32 

2.53 

FY94 

54.46 

33.8 

7.89 

1.32 

2.53 

FY95 

47.73 

38.8 

9.06 

1.51 

2.9 

FY96 

50 

35.2 

9.8 

2.2 

2.8 

FY97 

51.77 

34.45 

8.97 

1.2 

3.61 

FY98 

49.23 

36.6 

9.15 

2.18 

2.84 

FY99 

50.9 

38.69 

6.31 

2.34 

1.76 

FY00 

53.77 

38.6 

5.29 

1.17 

1.17 

FY01 

53.3 

38.41 

5.25 

1.17 

1.87 

FY02 

52.33 

36.1 

5.79 

3.93 

1.85 
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This  same  methodology  is  applied  to  the  F-18  E/F’s  TOA  in  Table  5.  Using  the  outlay 
rates  given  in  Table  3,  we  calculate  in  Table  4  that  $195.6  Million  of  FY92’s  TOA  will 
be  expended  during  FY92.  Furthermore,  $1 15.4  Million  of  FY92’s  TOA  will  be 
expended  during  FY93,  and  $27.9  Million  of  FY92’s  TOA  will  be  expended  during 
FY94.  Carrying  this  methodology  from  FY92  through  FY02  reveals  the  estimated  TY 
expenditures,  shown  in  Table  4’s  bolded  font.  Additionally,  Figure  8  compares  the 
distribution  of  TOA  and  estimated  expenditures.  Due  to  the  effect  of  multiyear  funding, 
expenditures  will  always  appear  to  “lag”  behind  TOA. 

Table  5:  F-18  E/F  TOA  to  Then  Year  Expenditures  Conversion 


Expenditures  (TY  $M) 

FY 

TOA 

FY92 

FY93 

FY94 

FY95 

FY96 

FY97 

FY98 

FY99 

FY00 

FY01 

FY02 

FY92 

349.5 

195.5 

115.4 

27.9 

5.0 

5.7 

0.0 

0.0 

- 

- 

- 

- 

FY93 

842.1 

- 

457.8 

285.3 

66.6 

11.1 

21.3 

0.0 

0.0 

- 

- 

- 

FY94 

1396.2 

_ 

_ 

760.4 

471.9 

110.2 

18.4 

35.3 

0.0 

0.0 

_ 

_ 

i 

& 

FY95 

1246 

- 

- 

- 

594.7 

483.4 

112.9 

18.8 

36.1 

0.0 

0.0 

- 

H 

FY96 

801.1 

- 

- 

- 

- 

400.6 

282.0 

78.5 

17.6 

22.4 

0.0 

1i3 

bh 

-a 

FY97 

345.4 

- 

- 

- 

- 

- 

178.8 

119.0 

31.0 

4.1 

12.5 

0.0 

T-> 

03 

FY98 

234.6 

- 

- 

- 

- 

- 

- 

115.5 

85.9 

21.5 

5.1 

6.7 

FY99 

195.6 

- 

- 

- 

- 

- 

- 

- 

99.6 

75.7 

12.3 

4.6 

FY00 

132.1 

- 

- 

- 

- 

- 

- 

- 

- 

71.0 

51.0 

7.0 

FY01 

13.9 

- 

- 

- 

- 

- 

- 

- 

- 

- 

7.4 

5.3 

FY02 

1.1 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

195.5 

573.1 

1073.5 

1138.2 

1011.0 

613.4 

367.1 

270.2 

194.7 

88.3 

24.1 

Following  the  application  of  outlay  rates,  the  F-18  E/F’s  estimated  then  year 
annual  expenditures  must  be  converted  to  base  year  expenditures  to  control  for  the  effects 
of  inflation.  As  shown  in  Table  6,  then  year  expenditures  are  divided  by  the  OSD- 
Comptroller’s  raw  inflation  index  to  obtain  base  year  dollars.  FY2014  was  selected  as 
the  base  year,  as  the  majority  of  our  research  was  completed  within  FY2014. 
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Figure  8:  F-18  E/F  Budget  vs.  TY  Expenditures  Profile 


Table  6:  F-18  E/F  Then-Year  Expenditures  to  Base-Year  Expenditures  Conversion 


Fiscal  Year 

1992 

1993 

1994 

1995 

1996 

1997 

1998 

1999 

2000 

2001 

2002 

TY  Expenditures  ($M) 

195.5 

573.1 

1073.5 

1138.2 

1011.0 

613.4 

367.1 

270.2 

194.7 

88.3 

24.1 

Inflation  Factor 

0.675 

0.693 

0.707 

0.721 

0.735 

0.750 

0.756 

0.762 

0.772 

0.786 

0.793 

BY  14  Expend.  ($M) 

289.7 

827.0 

1518.4 

1578.6 

1375.5 

817.9 

485.6 

354.6 

252.2 

112.3 

30.4 

Converting  Annual  Expenditures  to  Percent  Schedule  by  Percent  Expenditures 

Before  beginning  analysis,  the  estimated  expenditures  for  each  aircraft  program 
are  standardized  into  a  percent  schedule  by  percent  expenditure  format  by  applying 
consistent  development  start  and  end  criteria  to  each  program.  However,  the  selection  of 
start  and  end  criteria  for  development  should  not  be  made  indiscriminately,  as  Burgess 
(2006)  warns  that: 

Models. .  .cannot  be  more  accurate  than  the  underlying  cost  and  schedule 
estimates,  and  they  are  meaningful  only  when  the  scope  of  both  underlying 
estimates  (cost  and  schedule)  is  precisely  defined.  In  the  case  of  schedule 
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estimates,  models  must  have  precise  definitions  of  start  and  end  time,  indexed  to 
specific  programmatic  events  that  have  common  definitions  across  programs.  It 
doesn’t  help  to  have  an  accurate  model  that  predicts  60  percent  spent  at  50  percent 
time,  for  example,  if  the  definition  of  “time”  is  ambiguous  (Burgess,  2006:19). 
Therefore,  we  detennine  that  simply  using  the  annual  TOA  funding  or  outlay-rate 
estimated  expenditures  to  define  the  start  or  end  of  development  would  be  inappropriate, 
as  TOA  is  often  distorted  in  a  manner  that  disconnects  funding  from  development 
milestones.  For  example,  it  is  observed  that  many  historical  development  SAR  budgets 
do  not  distinguish  between  the  initial  and  follow-on  development  efforts;  specifically,  the 
occurrence  of  follow-on  development  results  in  multiple  funding  inflection  points  that 
could  significantly  skew  the  right  tail  of  the  S-curve.  It  is  detennined  that  these  follow- 
on  development  efforts  are  beyond  the  scope  of  the  initial  time  phased  estimate. 

Engineering  and  Manufacturing  (EMD)  contract  award  is  selected  as  our 
definition  of  schedule  start,  as  the  date  of  EMD  contract  award  is  commonly  published 
and  is  synonymous  with  the  primary  contractor  initiating  aircraft  development  efforts. 

For  most  development  efforts,  EMD  contract  award  coincides  with  the  published 
Milestone  B.  By  comparison,  Initial  Operating  Capability  (IOC)  is  selected  as  our 
definition  of  schedule  end  for  two  reasons.  First,  IOC  is  selected  as  development  end 
because  it  is  the  latest  milestone  event  that  is  consistently  reported  for  all  aircraft 
development  programs  within  our  database.  By  comparison,  Full  Operational  Capability 
(FOC)  was  also  considered  as  an  alternate  definition  of  schedule  end;  however,  it  is 
reported  inconsistently  as  only  four  of  26  programs  publish  FOC.  Second,  IOC  is 
selected  as  development  end  because  AFCAH  (2007)  reports  that  the  planned  IOC  date  is 
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highly  influential  on  time  phasing.  “The  IOC  is  a  primary  driver  in  detennining  the 
program’s  development  and  production  schedules. .  .the  program’s  time  phased  estimate 
therefore  must  be  consistent  with  the  schedule. .  .so  that  its  budgetary  inputs  can  support 
the  achievement  of  the  IOC”  (AFCAH,  2007:  15-10). 

Having  selected  appropriate  start  and  end  criteria  for  development,  we  next  must 
truncate  any  expenditures  that  occur  outside  of  development.  Linear  interpolation  is 
utilized  to  estimate  partial  years  which  result  from  EMD  contract  award  or  IOC  occurring 
during  the  fiscal  year  (any  date  other  than  30  September).  Following  truncation,  the 
standardization  to  percent  schedule  by  percent  expenditures  occurs,  as  shown  for  the  F- 18 
E/F  in  Figure  9.  This  procedure  is  repeated  for  all  26  aircraft  development  programs, 
until  a  dataset  of  standardized  percent  schedule  and  percent  expenditure  is  gathered.  The 
aggregated  database  generates  a  scatter  plot  with  a  visible  S-curve,  seen  in  Figure  10. 


j  Outlay  Rate-Estimated  Expenditures  (BY  14  $)  by  Fiscal  Year  [ 

FY1992 

FY1993 

FY1994 

FY1995 

FY1996 

FY1997 

FY1998 

289.70 

827.04 

1518.43 

1578.70 

1375.52 

817.90 

485.62 

FY1999 

FY2000 

FY2001 

FY2002 

FY2003 

FY2004 

FY2005 

354.55 

252.27 

112.37 

30.44 

7.63 

2.17 

0.36 

♦ 


j  Incremental  and  Cumulative  Expenditures  Controlling  for  RDTE  Start  and  End  | 

m 

EMD  Awd 

7/21/1992 

9/30/1992 

9/30/1992 

9/30/1993 

9/30/1993 

9/30/1994 

9/30/1994 

9/30/1995 

9/30/1995 

9/30/1996 

9/30/1996 

9/30/1997 

9/30/1997 

9/30/1998 

9/30/1998 

9/30/1999 

9/30/1999 

9/30/2000 

IOC 

9/30/2000 

9/15/2001 

Incremental 
Expenditures 
(BY  14$) 

56.35 

827.04 

1518.43 

1578.70 

1375.52 

817.90 

485.62 

354.55 

252.96 

107.75 

Cumulative 
Expenditures 
(BY  14$) 

56.35 

883.39 

2401.82 

3980.51 

5356.03 

6173.93 

6659.55 

7014.10 

7267.05 

7374.81 

♦ 


Percent  Schedule  by  Percent  Expenditures 


Schedule 

wma 

0.2396 

0.3488 

0.5675 

0.6766 

1.0000 

Expenditures 

■El 

m 

0.3257 

0.5397 

0.8372 

0.9030 

1.0000 

Figure  9:  F-18E/F  Standardization  to  Percent  Schedule  by  Percent  Expenditures 
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Figure  10:  Standardized  Percent  Schedule  by  Percent  Expenditures  Database 

Estimation  of  Weibull  and  Beta  Distribution  Parameters 

After  converting  each  program  to  a  percent  schedule  by  percent  expenditures 
format,  optimal  parameters  must  be  computed  for  each  program.  Nonlinear  estimation 
techniques  are  utilized  to  determine  the  best-fit  parameters  for  the  Rayleigh  (as  a  special 
case  of  the  Weibull),  Weibull,  and  Beta  distributions.  Analysis  is  completed  separately 
for  each  program  using  the  Microsoft  Excel  Solver  add-in,  with  the  parameter  cells 
identified  as  changing  cells  and  the  sum-squared  error  (SSE)  identified  as  the  target  cell. 
During  analysis,  Excel  Solver  allows  the  parameters  of  each  distribution  to  vary  until  the 
SSE  value  between  the  actual  expenditure  percentage  and  the  distribution-modeled 
expenditure  percentage  is  minimized.  As  an  example,  Table  7  shows  the  results  of 
optimal  parameter  analysis  on  the  F-18E/F  program,  using  the  Weibull  distribution. 
Changing  the  parameters  to  any  value  other  than  a  —  1.76171  and  /?  =  0.40402  would 
increase  error  between  actual  and  predicted  expenditures  for  the  F-18E/F.  The  estimated 
parameters  by  distribution  type  and  program  are  provided  in  Appendix  C. 
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Table  7:  F-18E/F  Minimized  SSE  for  the  Weibull  Distribution 


Cum. 

Schedule 

Cumulative  Expenditures 

Squared 

Error 

Actual 

Weibull  Predicted 

0 

0 

0 

0 

0.021238 

0.007641 

0.005560 

0.000004 

0.130422 

0.119784 

0.127533 

0.000060 

0.239605 

0.325678 

0.328570 

0.000008 

0.348789 

0.539744 

0.537844 

0.000004 

0.458271 

0.726260 

0.713077 

0.000174 

0.567454 

0.837165 

0.837861 

0.000000 

0.676638 

0.903013 

0.916301 

0.000177 

0.785821 

0.951089 

0.960382 

0.000086 

0.895304 

0.985389 

0.982794 

0.000007 

1.000000 

1.000000 

0.992819 

0.000052 

Sum  Squared  Error  = 

0.000572 

Parameters 

Alpha 

Beta 

1.76171 

0.40402 

Tests  for  Linear  Relationships  between  Parameters  and  Program  Characteristics 

Next,  using  linear  regression,  we  test  for  the  existence  of  relationships  between  an 
aircraft  development  program’s  attributes  and  its  estimated  parameters  for  the  Rayleigh, 
Weibull  and  Beta  models.  For  the  linear  regression  model,  the  distribution’s  parameter 
becomes  the  dependent  variable,  while  a  development  program’s  characteristics  are 
treated  as  independent  variables  (also  known  as  predictor  variables).  Based  on  previous 
S-curve  research,  it  is  postulated  that  the  following  program  attributes  are  significant 
predictor  variables:  development  program  length  (Brown  et  ah,  2002;  Burgess,  2006), 
development  expenditure  amount  (Dibbly,  1988),  branch  of  service  (Brown  et  ah,  2002), 
and  scheduled  time  to  first  flight  (Lee  et  ah,  1997).  In  addition,  several  other  program 
characteristics  are  considered  as  predictor  variables  for  our  linear  model,  as  summarized 
in  Table  8. 
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Table  8:  Program  Characteristics  Considered  as  Predictor  Variables 


# 

Predictor  Variables 

Previously  Documented 

1 

Length  of  Development  (EMD  Award  through  IOC) 

Brown  et  al.  (2002),  Burgess  (2006) 

2 

Total  Costs  of  Development  (BY14  Dollars) 

Dibbly  (1988) 

3 

Branch  of  Service 

Brown  et  al.  (2002) 

4 

Aircraft  Type  (Attack/Fighter,  Cargo,  Bomber,  or  1SR) 

- 

5 

Time  to  First  Flight  (%  Schedule) 

Lee  et  al.  (1997) 

6 

RDT&E  Prototypes 

- 

7 

Upgrade  Program  (Not  New  Start) 

- 

8 

Avionics-Specific  Upgrade  Program 

- 

9 

Budget  Threshold  Breach 

- 

10 

Schedule  Threshold  Breach 

- 

11 

Concurrency  (overlap  between  OT&E  and  Milestone  C) 

- 

Although  1 1  predictor  variables  are  considered  for  each  distribution  parameter,  it 
is  our  goal  to  make  each  model  as  parsimonious,  or  simplistic,  as  possible.  Therefore,  the 
final  linear  regression  for  each  distribution  parameter  will  be  limited  to  four  or  less 
predictor  variables,  with  a  goal  of  two  or  less.  There  are  several  reasons  we  strive  for 
parsimony.  First,  the  regression  equation  with  four  or  fewer  variables  is  less  effort  for 
cost  analysts  to  apply  and  decreases  the  risk  of  a  computational  error.  Second,  the  model 
with  fewer  variables  is  more  easily  explained  and  presented  to  program  managers  and 
other  key  decision  makers.  Third  (and  most  importantly),  DoD  cost  analyst  often  work 
with  relatively  small  sample  sizes  of  historic  program  cost  data.  An  often  cited  “10-to-l 
rule  of  thumb”  within  statistics  recommends  that  a  regression  should  have  10  or  more 
observations  per  independent  variable  (Vittinghoff  and  McCulloch,  2007).  Similarly,  a 
linear  modeling  textbook  by  Neter  et  al.  (1996)  states  that  a  ratio  of  six  to  10  predictor 
variables  per  predictor  variable  is  acceptable.  Due  to  having  only  26  useable  observations 
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within  this  thesis,  we  set  the  goal  of  2  or  fewer  predictor  variables  per  parameter,  with  an 
upper  limit  of  4  predictor  variables  per  parameter. 

Comparison  of  Final  Rayleigh,  Weibull,  and  Beta  S-curve  Models 

Using  the  linear  relationships  between  program  characteristics  and  parameters,  a 
final  model  will  be  developed  for  each  distribution  type.  Hence,  we  will  be  directly 
comparing  the  final  Rayleigh,  Weibull,  and  Beta  models,  with  the  intention  of  defining 
the  ‘best’  model.  To  determine  the  best  model,  the  predicted  TOA  profile,  derived  from 
predicted  annual  expenditures,  will  be  compared  against  the  actual  TOA  profile.  The 
best  model  will  be  defined  as  the  model  which  provides  for  the  highest  average  accuracy 
when  predicting  annual  TOA,  while  controlling  for  model  robustness  across  time  and 
aircraft  type.  Therefore,  we  next  present  the  Lee  et  al  (1997)  methodology  for  deriving  a 
TOA  profile  from  predicted  annual  expenditures,  followed  by  definitions  for  model 
accuracy  and  robustness. 

Convert  Annual  Expenditures  to  a  TOA  (Budget)  Profile 

The  first  step  in  this  process  of  converting  from  base  year  2014  annual 
expenditures  to  then  year  TOA  is  “escalating”  each  year’s  predicted  base  year  2014 
expenditures,  so  that  our  prior  controls  for  inflation  are  removed  and  current  year 
expenditures  are  obtained.  Escalation  is  accomplished  by  multiplying  each  year’s 
predicted  base  year  2014  expenditures  by  the  OSD-Comptroller’s  raw  inflation  index.  It 
should  be  noted  that  this  process  is  simply  a  reversal  of  the  current  year  to  base  year 
methodology  applied  earlier  in  Table  6.  The  second  step  in  our  process  involves 
transforming  our  predicted  current  year  expenditures  into  a  TOA  profile  through  a 
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application  of  the  OSD-Comptroller’s  outlay  rates.  We  elect  to  employ  Lee  et  al.’s 
(1997)  methodology  of  using  a  linear  system  of  equations  to  calculate  a  TOA  profile. 

Lee  et  al.’s  (1997)  methodology  defines  the  linear  equation 

Tfc  =  iPk  ~  s2^k-l  ~  s?Jk-2  ~  sjTk-j+ i)/si  (8) 

Where  t  represents  the  estimated  TOA  in  a  given  year,  s  represents  the  outlay  pattern 
value  (value  between  0  and  1),  and  0  represents  the  outlay,  also  known  as  annual 
expenditure. 

Model  Accuracy 

Our  research  will  utilize  the  Pearson  R7  to  judge  each  model’s  accuracy.  We 
make  the  distinction  that  R“  is  computed  separately  for  each  of  the  aircraft  development 
programs  and  then  averaged  to  arrive  at  a  mean  model  R“.  This  technique  ensures  that 
aircraft  development  programs  with  a  greater  number  of  years  of  RDTE  do  not  bias  the 
reported  R“  value;  hence,  each  aircraft  development  program  is  allotted  equal  weight  in 
detennining  the  model’s  accuracy. 

Robustness 

When  selecting  a  best  model,  consideration  will  also  be  given  to  model 
robustness.  As  documented  by  Stingier  (2010),  robustness  has  many  definitions  that  have 
changed  throughout  time.  However,  for  this  thesis,  we  will  comprise  a  simple  standard 
for  S-curve  model  robustness:  the  best  S-curve  model  should  be  able  to  accurately  predict 
annual  expenditures  (as  measured  by  R")  for  all  aircraft  types  across  all  time  periods.  For 
example,  an  S-curve  model  which  is  only  accurate  in  predicting  time-phased  expenditures 
for  1970’s-era  fighter  aircraft  would  be  useless  for  the  cost  analyst  attempting  to  time 
phase  a  contemporary  cargo  aircraft. 
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Summary 

This  chapter  explains  the  proposed  methodology  for  estimating  the  best  model  for 
time-phasing  aircraft  RDT&E  expenditures.  We  begin  by  first  obtaining  SAR  budget 
data  from  DAMIR  and  estimating  annual  expenditures  via  outlay  rates.  After  accounting 
for  inflation  through  the  application  of  the  OSD-Comptroller  inflation  rates,  the 
expenditure  data  are  transformed  into  a  percent  expenditure  by  percent  schedule  fonnat. 
This  transformation  is  necessary  to  establish  a  database  of  standardized  program  data  for 
analysis.  Next,  we  estimate  distribution  parameters  for  each  program  using  Excel’s 
Solver;  distribution  parameters  are  adjusted  until  the  SSE  is  minimized.  Following  the 
estimation  of  parameters,  linear  relationships  between  the  distribution  parameters  and 
program  characteristics  are  developed.  These  linear  relationships  are  then  used  to 
construct  our  final  Rayleigh,  Weibull,  and  Beta  models.  Finally,  using  the  R2  for  annual 
TOA  and  robustness  checks,  these  final  models  are  compared  and  a  “best”  model  is 
defined.  In  the  next  chapter,  we  present  the  analysis  of  our  data  and  model  development. 
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IV.  Analysis  and  Results 


Introduction 

This  chapter  provides  the  results  from  the  methodology  outlined  in  Chapter  3. 
First,  using  our  database  of  26  aircraft,  we  calculate  the  average  percent  of  expenditures 
at  the  development  program  midpoint  to  detennine  the  accuracy  of  the  60  percent 
expenditures  by  50  percent  schedule  “rule  of  thumb”.  Second,  linear  relationships 
between  development  program  characteristics  and  distribution  parameters  are  developed. 
Finally,  using  the  developed  linear  relationships,  the  best  Rayleigh,  Weibull,  and  Beta 
models  are  compared.  The  models  are  judged  by  their  R2  values  in  addition  to  robustness 
across  time,  aircraft  type,  and  upgrade  characteristic. 

Accuracy  of  60/40  “Rule  of  Thumb” 

As  introduced  earlier,  the  60  percent  expenditures  at  50  percent  schedule  heuristic 
(also  known  as  the  60/40  rule  of  thumb)  is  well  known  to  cost  analysts  (Lee  at  ah,  1997). 
To  test  the  accuracy  of  this  rule,  we  calculate  the  estimated  percent  expenditures  at  50 
percent  schedule  for  the  26  aircraft  development  programs  in  our  database  through  linear 
interpolation.  We  discover  that  programs  expend  only  56.3  percent  of  their  total 
expenditures  at  50  percent  schedule,  on  average,  as  summarized  in  Appendix  B.  More 
specifically,  7  programs  expend  more  than  60  percent  by  the  development  midpoint,  16 
expend  less  than  60  percent  by  the  program  midpoint,  and  3  expend  approximately  60 
percent  by  the  program  midpoint. 

However,  a  noticeable  time  trend  emerges  when  programs  are  plotted  across  time 
in  Figure  11.  Expenditures  for  aircraft  development  programs  have  generally  become 
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more  “front-loaded”  over  time,  with  a  greater  percentage  of  expenditures  occurring  by 
the  development  midpoint.  The  most  significant  change  in  development  program 
spending  is  visible  during  the  1980s,  the  period  during  which  the  President’s  Blue  Ribbon 
Commission  on  Defense  Management  was  preparing  its  final  report.  The  report 
recommended  sweeping  changes  to  acquisitions  management,  to  include  increased  usage 
of  prototyping,  limitations  on  concurrency,  early  development  testing,  and  a  multi-year 
appropriations  cycle  (Blue  Ribbon  Commission,  1986).  We  hypothesize  that  the 
President’s  Blue  Ribbon  Panel  had  an  influence  on  the  distribution  of  aircraft 
development  expenditures,  and  therefore  elect  to  analyze  separately  the  14 
“contemporary”  programs  which  initiated  development  during  the  last  30  years.  If  the 
contemporary  programs  (from  1985  and  later)  are  analyzed  in  isolation,  we  discover  that 
programs  expend  a  mean  of  63. 13  percent  of  total  expenditures  at  50  percent  schedule, 
with  a  median  of  59.83  percent  expenditures  at  50  percent  schedule. 

Additionally,  a  second  trend  emerges  between  contemporary  upgrade  (to  include 
modification  programs)  and  contemporary  new  start  programs,  as  shown  in  Table  9.  On 
average,  we  find  that  the  expenditures  for  contemporary  upgrade  programs  are  more 
“front  loaded”  than  their  contemporary  new  start  counterparts.  Contemporary  upgrade 
programs  expend  65.27  percent  of  total  expenditures  at  50  percent  schedule,  while 
contemporary  new  start  programs  expend  55.04  percent  of  total  expenditures  at  50 
percent  schedule.  As  a  result,  we  assess  that  the  60/40  rule  is  generally  not  an  accurate 
heuristic  for  the  contemporary  aircraft  development  program,  given  that  the  analyst  can 
identify  the  development  program  as  either  a  new  start  or  upgrade. 
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Figure  11:  Percent  Expenditures  at  50%  Schedule,  by  Year  of  Contract  Award 


Table  9:  Percent  Expenditures  at  50%  Schedule,  New  Start  vs.  Upgrade  (1985-later) 


New  Start 
Program 

Percent 

Expenditures  at 
50%  Schedule 

F-22 

0.60 

F-35  AF 

0.60 

JSTARS 

0.54 

P-8A 

0.46 

Mean: 

0.55 

Median: 

0.57 

Upgrade 

Program 

Percent 

Expenditures  at 
50%  Schedule 

AWACS  BLOCK  40 

0.69 

AWACS  RSIP 

0.80 

B-l  CMUP 

0.49 

B-2  EHF  1 

0.75 

B-2  RMP 

0.60 

C-5  AMP 

0.57 

C-5  RERP 

0.76 

E-2D 

0.70 

EA-18G 

0.49 

EA-6B  ICAP 

0.56 

F-18E/F 

0.77 

Mean: 

0.65 

Median: 

0.69 
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Linear  Relationships  between  Parameters  and  Program  Characteristics 

Utilizing  unique  program  characteristics,  linear  relationships  are  developed  to 
explain  variation  in  the  estimated  distribution  parameter  values.  In  addition  to  the  1 1 
program  characteristics  previously  documented,  we  consider  interactions  between 
predictor  variables  and  transfonnations  of  predictor  variables,  such  as  the  application  of 
natural  log,  to  increase  the  explanatory  power  of  our  models.  Predictor  variables  which 
are  not  statistically  significant  at  the  5  percent  level  are  excluded  from  the  model  outputs. 
Based  on  the  discovery  of  the  time  trend  observed  earlier,  we  also  consider  the  inclusion 
of  an  indicator  variable,  also  known  as  a  “dummy  variable”,  for  all  aircraft  with  an  EMD 
contract  award  date  before  1985. 

Additionally,  it  should  be  emphasized  that  the  C-17A  is  excluded  from  the  linear 
regression  models  for  two  reasons.  First,  the  C-17A  is  an  overly  influential  data  point 
across  all  distribution  parameters  and  consistently  resulted  in  Cook’s  D  values  above  1. 

A  Cook’s  D  value  above  0.5  indicates  that  a  data  point  is  significantly  influential  on  the 
model  estimates  (Neter,  Kutner,  Nachtsheim,  and  Wasserman,  1996:  380).  Second,  a 
qualitative  review  of  the  acquisition  history  of  the  C-17A  reveals  that  the  C-17A  aircraft 
suffered  extensive  development  delays  due  to  contracting  difficulties,  budget  cuts,  and 
management  problems,  as  evidenced  by  a  1990  USAF  Inspector  General  investigation  for 
management  improprieties  (Saxer,  1995).  Although  schedule  delays  are  not  uncommon 
for  aircraft  development,  we  detennine  that  the  extent  of  the  C-17A’s  delays  make  it  a 
unique  data  point  that  should  be  separated  from  other  aircraft  within  our  sample. 
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Rayleigh  Distribution  -  Scale  Parameter  P 

Three  predictor  variables  are  significant  at  explaining  variation  in  the  Rayleigh’s 
scale  parameter  [3:  time  to  first  flight,  upgrade  program,  and  the  interaction  between  the 
pre-1985  indicator  and  upgrade.  Together,  these  three  variables  provide  a  R"  of  0.535, 
and  significance  at  the  5  percent  level  is  maintained  for  all  variables,  as  displayed  in 
Table  10.  Next,  we  test  for  influential  data  points  by  utilizing  the  Cook’s  D  statistic.  As 
shown  in  Appendix  D,  the  absence  of  values  over  0.5  indicates  that  no  influential 
observations  exist  for  the  linear  regression  model.  Finally,  we  use  the  residuals,  which 
are  the  difference  between  the  observed  and  predicted  parameter  values,  to  test  separately 
for  nonnality  and  constant  variance.  Appendix  D  displays  the  normal  distribution  fit  to 
residual  values;  the  Shapiro-Wilks  test  p-value  greater  than  0.05  indicates  that  the 
residual  values  are  normally  distributed.  Residuals  appear  to  have  a  reasonably  uniform 
distribution  across  the  predicted  range,  showing  constant  variance.  Constant  variance  is 
further  confirmed  by  the  Breusch-Pagan  test;  the  reported  p-value  of  0.5264  fails  to 
rejects  the  null  hypothesis  of  constant  variance. 

Weibull  Distribution  -  Shape  Parameter  a 

One  predictor  variable  is  significant  at  explaining  variation  in  the  Weibull’s  shape 
parameter  a:  the  pre-1985  indicator.  As  shown  in  Table  11,  this  single  variable  provides 
a  R"  of  0.357,  with  statistical  significance  at  the  0.05  level.  As  shown  in  Appendix  D,  the 
absence  of  Cook’s  D  values  over  0.5  indicates  that  no  influential  observations  exist  for 
either  linear  regression  model.  Additionally,  Appendix  D  displays  the  normal 
distribution  fit  to  residual  values;  the  Shapiro-Wilk’s  p-value  greater  than  0.05  indicates 
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that  the  residual  values  are  normally  distributed.  Residuals  appear  to  be  unifonnly 
distributed,  with  a  Breusch-Pagan  p-value  of  0.69 1 1 . 

Table  10:  Summary  of  Fit  -  Rayleigh  Scale  Parameter  P 


Summary  of  Fit 

R2 

0.534592 

R2-  Adjusted 

0.468105 

Model  Parameter  Estimates 

Term 

Estimate 

Std.  Error 

Prob.  >  |t| 

Std.  Beta 

V1F 

Intercept 

0.450261 

0.048829 

<0.0001 

- 

- 

First  Flight  (%  Schedule) 

0.240399 

0.100000 

0.0255 

0.36211 

1.02381 

Upgrade  Program 

-0.078185 

0.025229 

0.0054 

-0.48676 

1.11316 

Interaction 

0.148797 

0.045965 

0.0039 

0.50304 

1.08960 

Table  11:  Summary  of  Fit  --  Weibull  Shape  Parameter  a 


Summary  of  Fit 

R2 

0.356652 

R2-  Adjusted 

0.328681 

Model  Parameter  Estimates 

Temi 

Estimate 

Std.  Error 

Prob.  >  |t| 

Std.  Beta 

V1F 

Intercept 

1.91047 

0.049748 

<0.0001 

- 

- 

Pre-1985  Contract  Award 

0.503002 

0.140866 

0.0016 

0.597204 

1 

Weibull  Distribution  -  Scale  Parameter  P 

As  an  indicator  of  robustness,  it  is  discovered  that  the  Weibull’s  scale  parameter  P 
is  influenced  by  the  same  predictor  variables  as  the  Rayleigh’s  scale  parameter  P:  time  to 
first  flight,  upgrade  program,  and  the  interaction  between  the  pre-1985  indicator  and 
upgrade.  Together,  these  three  variables  provide  a  R2  of  0.547,  and  statistical 
significance  is  maintained  at  the  0.05  level  for  all  variables,  as  displayed  in  Table  12.  As 
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shown  in  Appendix  D,  the  absence  of  Cook’s  D  values  over  0.5  indicates  that  no 
influential  observations  exist  for  the  linear  regression  model.  Additionally,  Appendix  D 
displays  the  normal  distribution  fit  to  residual  values;  the  Shapiro-Wilk’s  p-value  greater 
than  0.05  indicates  that  the  residual  values  are  normally  distributed.  Residuals  appear  to 
be  unifonnly  distributed,  with  a  Bruesch-Pagan  p-value  of  0.5 176. 


Table  12:  Summary  of  Fit  --  Weibull  Scale  Parameter  P 


Summary  of  Fit 

R2 

0.546873 

R2-  Adjusted 

0.482141 

Model  Parameter  Estimates 

Term 

Estimate 

Std.  Error 

Prob.  >  |t 

Std.  Beta 

V1F 

Intercept 

0.442652 

0.049748 

<0.001 

- 

- 

First  Flight  (%  Schedule) 

0.258704 

0.101884 

0.0191 

0.377406 

1.0238 

Upgrade  Program 

-0.079064 

0.025703 

0.0057 

-0.47673 

1.1132 

Interaction 

0.156363 

0.04683 

0.0031 

0.51196 

1.0896 

Beta  Distribution  -  Shape  Parameter  a 

Three  predictor  variables  are  significant  at  explaining  variation  in  the  Beta’s 
shape  parameter  a:  the  pre-1985  indicator,  length  of  development,  and  the  natural  log  of 
1  divided  by  the  length  of  development.  Table  13  displays  that  these  variables  together 
provide  a  R“  of  0.462,  and  significance  at  the  5  percent  level  is  maintained  for  all 
variables.  As  shown  in  Appendix  D,  the  absence  of  Cook’s  D  values  over  0.5  indicates 
that  no  influential  observations  exist  for  the  regression  model.  Additionally,  Appendix  D 
displays  the  normal  distribution  fit  to  residual  values;  the  Shapiro-Wilk’s  p-value  greater 
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than  0.05  indicates  that  the  residual  values  are  normally  distributed.  Residuals  appear  to 
be  uniformly  distributed,  with  a  Bruesch-Pagan  p-value  of  0.6473. 

Table  13:  Summary  of  Fit  -  Beta  Shape  Parameter  a 


Summary  of  Fit 

R2 

0.462269 

R2-  Adjusted 

0.385451 

Model  Parameter  Estimates 

Std. 

Term 

Estimate 

Error 

Prob.  >  |t| 

Std.  Beta 

V1F 

Intercept 

-1.803242 

0.362462 

0.0988 

- 

- 

Pre-1985  Contract  Award 

0.742528 

0.873358 

0.0183 

0.745497 

1.67176 

Length  of  Development 

-0.324556 

0.112016 

0.0086 

-2.15116 

21.5268 

Ln(  1/Length  of  Develop.) 

-2.945843 

0.91015 

0.0040 

-2.51231 

23.5293 

Beta  Distribution  -  Shape  Parameter  6 

Four  predictor  variables  are  useful  in  explaining  variation  in  the  Beta’s  shape 
parameter  P:  the  pre-1985  indicator,  length  of  development,  upgrade,  and  the  interaction 
between  upgrade  and  the  pre-1985  indicator.  Together,  these  variables  provide  a  R“  of 
0.593,  and  significance  at  the  5  percent  level  is  maintained  for  all  variables,  as  displayed 
in  Table  14.  As  shown  in  Appendix  D,  the  absence  of  Cook’s  D  values  over  0.5  indicates 
that  no  influential  observations  exist  for  the  regression  model.  Additionally,  Appendix  D 
displays  the  nonnal  distribution  fit  to  residual  values;  the  Shapiro-Wilk’s  p-value  greater 
than  0.05  indicates  that  the  residual  values  are  normally  distributed.  The  residuals  plot 
appears  to  be  unifonnly  distributed  with  a  reported  Breusch-Pagan  p-value  of  0.419. 
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Table  14:  Summary  of  Fit  -  Beta  Shape  Parameter  P 


Summary  of  Fit 

R2 

0.593305 

R2-  Adjusted 

0.511966 

Model  Parameter  Estimates 

Std. 

Temi 

Estimate 

Error 

Prob.  >  |t| 

Std.  Beta 

V1F 

Intercept 

-0.361293 

0.548769 

0.5178 

- 

- 

Pre- 1985  Contract  Award 

1.426644 

0.377201 

0.0012 

1.11796 

4.29664 

Length  of  Development 

0.155172 

0.039521 

0.0008 

0.80274 

2.05559 

Upgrade  Program 

1.492916 

0.306782 

<0001 

1.19306 

2.95581 

Interaction 

-1.646058 

0.45711 

0.0018 

-0.71431 

1.93505 

Functional  Form  of  Final  Models 

Having  established  program  characteristics  to  operate  as  predictor  variables  for 
the  distribution  parameters,  we  write  the  final  functional  forms  of  our  Rayleigh,  Weibull, 
and  Beta  non-linear  equations.  Additionally,  we  identify  a  baseline  model  for 
comparison:  the  60/40  Rayleigh  model,  provided  in  Lee  et  al.  (1997).  It  should  be 
emphasized  that  the  Lee  et  al.  60/40  model  controls  for  the  Rayleigh’s  infinite  tail  by 
assuming  that  schedule  completion  occurs  at  97  percent  of  expenditures;  therefore,  the 
right  side  of  the  given  equation  is  divided  by  0.97. 

Final  Rayleigh  Distribution  Model 

°/o  schedule .  2 

%  Expenditures  —  1  —  e  P  (9) 

/?  =  0.45026  +  0.2404  (%  Schedule  at  First  Flight ) 

—  0.07819  ( Upgrade )  +  0.1488  (Pre  —  1985  x  Upgrade ) 
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Final  Weibull  Distribution  Model 


scrieuuie-.  a 

%  Expenditures  =  1  —  e  P 
a  =  1.91047  +  0.503  (Pre  -  1985) 

/?  =  0.44265  +  0.2587  (%  Schedule  at  First  Flight ) 

—  0.07906  ( Upgrade )  +  0.15636  (Pre  —  1985  x  Upgrade ) 
Pina/  Beta  Distribution  Model 


%  Expenditures 


(l-%  schedule)P  1(%  schedule)01  1r(a+(i) 


r(a)r(p) 

a  =  -1.80324  +  0.74253  ( Pre  -  1985)  -  0.32456  (Length)  - 
2.94584  (In  — ^— ) 

Length ' 

P  =  -0.36129  +  1.42664  (Pre  -  1985)  +  0.155517  (Length) 
+  1.49292  (Upgrade)  —  1.64606  (Pre  —  1985  x  Upgrade) 
Baseline  60/40  “Rule  of  Thumb”  Model 


%  Expenditures 


l_£-3.52(%  schedule )2 
0.97 


(10) 


(ii) 


(12) 


Comparison  between  Final  Models’  Predictive  Capability 
Annual  TOA  R2 

Next,  using  our  written  equations,  we  compare  the  predictive  accuracy  of  our 
Rayleigh,  Weibull,  and  Beta  models  against  the  baseline  60/40  model.  As  previously 
explained  in  Chapter  3,  accuracy  is  defined  as  R",  or  the  percentage  of  total  variability  in 
annual  TOA  that  is  explained  by  the  model.  After  applying  our  methodology  for 
calculating  R“,  we  find  that  the  Weibull  model  offers  the  highest  nominal  predictive 
ability  across  all  programs,  on  average,  as  shown  in  Table  15.  When  only  contemporary 
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programs  with  a  contract  award  of  1985  or  later  are  considered,  the  Weibull  again 
outperfonns  the  Rayleigh  and  Beta  distributions,  although  by  a  less  significant 
percentage.  As  a  result,  we  are  not  yet  satisfied  the  Weibull  model  is  superior,  and  next 
turn  to  robustness  checks  to  validate  if  any  model  is  consistently  superior  (or  inferior)  for 
predicting  a  particular  time  period,  aircraft  type  or  upgrade  program. 


Table  15:  R2  Comparison  between  Final  Models 


60/40 

Rayleigh 

Weibull 

Beta 

Mean  R2 

0.6807 

0.7365 

0.7463 

0.6988 

Mean  R2  (CY85  -14  contract  award  only) 

0.6813 

0.7149 

0.7152 

0.6648 

Median 

0.7240 

0.8002 

0.8011 

0.7163 

Max  R2 

0.9488 

0.9538 

0.9707 

0.9138 

Min  R2 

0.2785 

0.3173 

0.3715 

0.3784 

R2  Range  (All  Historical  Data) 

60/40 

Rayleigh 

Weibull 

Beta 

<0.5 

4 

3 

1 

2 

0.5  <0.6 

4 

3 

5 

5 

0.6  <0.7 

4 

4 

2 

4 

0.7  <0.8 

7 

1 

4 

7 

0.8  <0.9 

4 

10 

9 

6 

0.9  <  1.0 

3 

4 

4 

1 

Total  25  25  25  25 


Robustness  Check  1 

As  our  first  test  of  robustness,  we  examine  each  model’s  predictive  ability  as  a 
function  of  time  in  Figure  12  and  Table  16.  Although  it  is  observed  that  no  model  is 
radically  superior  for  predicting  budget  requirements  for  more  contemporary  aircraft,  the 
Weibull  model  holds  as  the  most  predictive  during  two  of  the  five  10  year  periods. 
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Figure  12:  Models'  Predictive  Ability  over  Time 


Table  16:  Models’  Predictive  Ability  over  Time 


Year  of  EMD 
Contract  Award 

Sample 

(n) 

60/40 

Rayleigh 

Weibull 

Beta 

CY65-74 

4 

0.7684 

0.8137 

0.7149 

0.7466 

CY75-84 

6 

0.5583 

0.7012 

0.8200 

0.7545 

CY85-94 

4 

0.5679 

0.6489 

0.6737 

0.7006 

CY95-04 

10 

0.7223 

0.7291 

0.7182 

0.6367 

CY05-14 

1 

0.7240 

0.8359 

0.8504 

0.8024 

Robustness  Check  2 

Next,  as  a  second  check  of  robustness,  the  mean  R  value  for  each  aircraft  type  is 
computed.  Aircraft  are  grouped  into  four  broad  development  categories:  attack/fighter, 
bomber,  cargo,  and  ISR/electronic.  For  both  fighter  (Table  17)  and  bomber  (Table  19) 
aircraft,  it  is  observed  that  the  Weibull  model  generally  outperforms  both  the  Rayleigh 
and  Beta  model,  as  measured  by  the  mean.  Conversely,  for  cargo  (Table  18),  the  Beta 
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outperforms  the  Rayleigh  and  Weibull;  however,  with  a  sample  size  of  only  two  cargo 
aircraft,  our  inference  is  limited.  Finally,  it  is  seen  in  Table  20  that  we  are  least  successful 
at  predicting  the  annual  TOA  for  ISR/Electronic  aircraft,  as  the  best  performing  model 
(the  Rayleigh)  offers  a  mean  R  of  only  0.65. 


Table  17:  Attack/Fighter  Aircraft  R2  Comparison 


Program 

60/40 

Rayleigh 

Weibull 

Beta 

A- 10 

0.9182 

0.8836 

0.7053 

0.5404 

F-14A 

0.9488 

0.9538 

0.8011 

0.7761 

F-14D 

0.6007 

0.6944 

0.8473 

0.6825 

F-15A 

0.3543 

0.4983 

0.5537 

0.7993 

F-16A 

0.9327 

0.9293 

0.9707 

0.7394 

F-18  E/F 

0.6796 

0.8806 

0.8937 

0.7479 

F-18A 

0.2785 

0.3173 

0.5118 

0.7160 

F-22 

0.8109 

0.7944 

0.8314 

0.8716 

F-35  (USAF) 

0.8082 

0.8077 

0.8167 

0.5375 

F-5E 

0.8523 

0.9191 

0.7996 

0.8708 

Mean: 

0.7184 

0.7678 

0.7731 

0.7281 

Median: 

0.8096 

0.8442 

0.8089 

0.7436 

Table  18:  Cargo  Aircraft  R2  Comparison 


Program 

60/40 

Rayleigh 

Weibull 

Beta 

C-5  AMP 

0.7070 

0.5679 

0.5261 

0.6357 

C-5  RERP 

0.7384 

0.9007 

0.9316 

0.8476 

Mean 

0.7227 

0.7343 

0.7288 

0.7417 

Median: 

0.7227 

0.7343 

0.7288 

0.7417 
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Table  19:  Bomber  Aircraft  R2  Comparison 


Program 

60/40 

Rayleigh 

Weibull 

Beta 

B-l  CMUP 

0.8287 

0.8202 

0.7880 

0.7283 

B-2  EHF  1 

0.7240 

0.8359 

0.8504 

0.8024 

B-2RMP 

0.7750 

0.8000 

0.8044 

0.7163 

B-2  A 

0.5837 

0.8002 

0.9048 

0.5974 

Mean 

Median: 

0.7278 

0.7495 

0.8141 

0.8102 

0.8369 

0.8274 

0.7111 

0.7223 

Table  20:  ISR/Electronic  Aircraft  R2  Comparison 


Program 

60/40 

Rayleigh 

Weibull 

Beta 

AWACS  Block  40 

0.5065 

0.6728 

0.7139 

0.6058 

AWACS  RS1P 

0.4111 

0.5664 

0.5980 

0.6919 

E-2D 

0.7726 

0.8633 

0.8605 

0.8190 

E-6A 

0.7323 

0.8703 

0.8808 

0.9138 

EA-18G 

0.6989 

0.5373 

0.5004 

0.5630 

EA-6B  1CAP 

0.7307 

0.6761 

0.6256 

0.5353 

JSTARS 

0.3700 

0.3543 

0.3715 

0.4910 

P-8A 

0.6575 

0.6452 

0.6151 

0.3784 

Mean 

0.6099 

0.6482 

0.6457 

0.6248 

Median: 

0.6782 

0.6590 

0.6203 

0.5844 

Robustness  Check  3 

As  a  third,  and  final,  check  of  robustness,  the  ability  of  each  model  to  predict  annual 
TOA  for  both  “new  starts”  and  “upgrade”  programs  is  compared.  In  Table  21  and  Table 
22,  it  is  shown  that,  on  average,  we  predict  the  distribution  of  TOA  for  new  starts  and 
upgrade  programs  with  equal  success.  Across  both  new  starts  and  upgrades,  the  Weibull 
model  generally  outperforms  the  Rayleigh  and  Beta  models. 
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Table  21:  New  Start  Aircraft  R2  Comparison 


Program 

60/40 

Rayleigh 

Weibull 

Beta 

A-10 

0.9182 

0.8836 

0.7053 

0.5404 

B-2A 

0.5837 

0.8002 

0.9048 

0.5974 

E-6A 

0.7323 

0.7803 

0.8808 

0.9138 

F-14A 

0.9488 

0.9538 

0.8011 

0.7761 

F-15A 

0.3543 

0.4983 

0.5537 

0.7993 

F-16A 

0.9327 

0.9293 

0.9707 

0.7394 

F-18A 

0.2785 

0.3173 

0.5118 

0.7160 

F-22 

0.8109 

0.7944 

0.8314 

0.8716 

F-35  AF 

0.8082 

0.8077 

0.8167 

0.5375 

J  STARS 

0.3700 

0.3543 

0.3715 

0.4910 

P-8A 

0.6575 

0.6452 

0.6151 

0.3784 

T-46A 

0.5962 

0.8339 

0.9552 

0.8628 

Mean: 

0.6659 

0.7232 

0.7432 

0.6853 

Median: 

0.6949 

0.8040 

0.8089 

0.7277 

Table  22:  Upgrade  Aircraft  R2  Comparison 


Program 

60/40 

Rayleigh 

Weibull 

Beta 

AWACS  Block  40 

0.5065 

0.6728 

0.7139 

0.6058 

AWACS  RS1P 

0.4111 

0.5664 

0.5980 

0.6919 

B-l  CMUP 

0.8287 

0.8202 

0.7880 

0.7283 

B-2  EHF  1 

0.7240 

0.8359 

0.8504 

0.8024 

B-2  RMP 

0.7750 

0.8000 

0.8044 

0.7163 

C-5  AMP 

0.7070 

0.5679 

0.5261 

0.6357 

C-5  RERP 

0.7384 

0.9007 

0.9316 

0.8476 

E-2D 

0.7726 

0.8633 

0.8605 

0.8190 

EA-18G 

0.6989 

0.5373 

0.5004 

0.5630 

EA-6B  1CAP 

0.7307 

0.6761 

0.6256 

0.5353 

F-14D 

0.6007 

0.6944 

0.8473 

0.6825 

F-18E/F 

0.6796 

0.8806 

0.8937 

0.7479 

F-5E 

0.8523 

0.9191 

0.7996 

0.8708 

Mean: 

0.6943 

0.7488 

0.7492 

0.7113 

Median: 

0.7240 

0.8000 

0.7996 

0.7163 
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Summary 

Chapter  4  provides  the  results  of  applying  our  methodology  to  a  sample  of  26 
ACAT  1  aircraft  development  programs.  First,  it  is  shown  that  the  accuracy  of  the  60/40 
“rule  of  thumb”  is  limited,  as  it  does  not  account  for  the  differences  in  time  phasing 
between  new  start  and  upgrade  development  programs.  Next,  we  provide  goodness-of-fit 
results  that  show  that  three  unique  aircraft  development  characteristics  may  be  used  to 
estimate  the  Rayleigh,  Weibull,  and  Beta  distribution  parameters.  Specifically,  it  is 
shown  that  time  to  first  flight,  length  of  development,  and  upgrade  are  statistically 
significant  at  the  0.05  level  for  estimating  one  or  more  distribution  parameters.  Finally, 
we  use  the  estimated  linear  relationships  to  construct  a  final  Rayleigh,  Weibull,  and  Beta 
model.  Through  a  series  of  accuracy  and  robustness  checks,  these  final  models  are 
compared  against  the  baseline  60/40  model  provided  by  Lee  et  al.  (1997).  We  discover 
that  the  Weibull  model  outperfonns  other  proposed  models  for  the  majority  of  robustness 
checks.  In  the  following  chapter,  Chapter  5,  we  conclude  our  research  effort  by 
answering  Chapter  1  research  questions,  recognizing  model  limitations,  and  identifying 
areas  for  future  research. 
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V.  Conclusions  and  Recommendations 


Introduction 

In  this  chapter,  we  revisit  our  initial  research  questions  to  validate  that  our 
research  accomplished  its  intended  goal.  Additionally,  we  review  the  limitations  of 
findings,  identify  areas  for  future  research,  and  conclude  by  summarizing  the  significance 
of  this  research. 

Research  Questions  Answered 

1  -  Is  the  ‘ rule  of  thumb  ’  that  60  percent  of  expenditures  occur  by  50  percent 
schedule  (60/40  S-curve)  accurate  for  contemporary  aircraft  development  programs? 

Lee  et  al.  (1997)  trace  the  origin  of  the  60/40  rule  of  thumb  to  a  circa- 1980 
aircraft  development  study  completed  by  the  OSD  Cost  Analysis  Improvement  Group 
(CAIG).  However,  the  OSD-CAIG  study  could  not  be  recovered,  so  we  instead  review 
all  available  studies  on  time  phasing  aircraft  development. 

During  the  1970’s,  both  Weida  (1976)  and  General  Research  Corporation  (1979) 
find  that  aircraft  development  contracts  have  57  percent  expenditures  at  50  percent 
schedule.  Using  the  Weibull  parameters  located  in  the  appendix  from  Brown  (2001),  we 
separately  estimate  that  aircraft  development  programs  have  57.1  percent  expenditures  at 
50  percent  schedule,  on  average.  Similarly,  the  26  historical  aircraft  development 
programs  from  this  thesis  report  56.3  percent  expenditures  at  50  percent  schedule,  on 
average.  These  results  would  likely  lead  the  cost  analyst  to  erroneously  select  a  56/44  or 
57/43  “rule  of  thumb”  over  the  60/40. 
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However,  attempting  to  identify  a  single  rule  of  thumb  that  summarizes  all 
aircraft  across  time  would  be  a  mistake,  as  two  significant  trends  emerge.  Firstly,  as 
shown  earlier  in  Figure  13,  a  significant  time  trend  is  observed  within  our  data.  Aircraft 
programs  which  began  development  during  calendar  year  1985  or  later  have  expenditure 
distributions  which  are  more  “front  loaded”  (skewed  right)  than  programs  from  before 
1985.  As  a  result,  when  the  14  “contemporary”  programs  from  1985  and  later  are 
analyzed  in  isolation,  we  find  that  programs  expend  a  mean  of  63. 1  percent  of  total 
expenditures  at  50  percent  schedule,  with  a  median  of  59.8  percent  expenditures  at  50 
percent  schedule. 

Next,  we  further  delineate  contemporary  aircraft  as  either  new  starts  or  upgrades 
in  Table  9.  It  is  discovered  that  contemporary  new  starts  (n=4)  expend  a  mean  of  55.0 
percent  expenditures  at  50  percent  schedule,  while  contemporary  upgrades  (n=l  1)  expend 
a  mean  of  65.3  percent  expenditures  at  50  percent  schedule.  Therefore,  as  a  heuristic,  we 
detennine  that  the  60/40  rule  of  thumb  generally  overestimates  early  expenditures  for 
new  starts,  while  underestimating  early  expenditures  for  upgrade  programs. 

2  -  What  program  and/or  schedule  characteristics  best  predict  distribution 
parameters  for  the  Rayleigh,  Weibull,  and  Beta  distributions? 

As  summarized  previously  in  Table  8,  eleven  unique  attributes  were  considered 
initially  as  predictor  variables  for  the  distribution  parameters.  However,  only  three  of 
eleven  characteristics  were  statistically  significant.  The  time  to  first  flight  and  program 
upgrade  characteristics  are  predictive  for  the  Rayleigh  and  Weibull  distribution 
parameters,  while  the  time  to  first  flight,  upgrade,  length  of  development  characteristic 
are  predictive  for  the  Beta  distribution  parameters.  Total  development  costs,  branch  of 
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service,  aircraft  type,  prototyping,  budget/schedule  threshold  breaches,  and  concurrency 
were  not  statistically  significant  predictor  variables  (at  the  0.05  level)  for  any  of  the 
distribution  parameters. 

3  -  Which  distribution  (Rayleigh,  Weibull,  or  Beta)  provides  the  best  S-curve 

model  for  time  phasing  contemporary  aircraft  development  programs? 

The  Weibull  distribution  model,  which  explained  74.6  percent  of  the  total  variation 
in  annual  expenditures,  is  marginally  more  accurate  than  our  Rayleigh  and  Beta  models, 
which  explained  73.7  percent  and  69.9  percent  of  variation,  respectively.  All  three 
models  proved  more  accurate  than  the  60/40  model,  which  explained  only  68.0  percent  of 
total  variation.  Robustness  checks  reveal  that  the  Weibull  model  is  superior  across  most 
aircraft  types  and  sampled  time  periods,  with  exceptions. 

Due  to  these  exceptions,  we  recognize  that  no  time  phasing  model  is  perfect,  and 
recommend  that  the  cost  analyst  consider  the  application  of  two  or  more  S-curve  models 
as  a  “cross  check”  of  their  time  phased  estimate.  To  facilitate  this  cross-check,  we  again 
provide  the  final  functional  forms  for  our  Rayleigh,  Weibull,  and  Beta  models  in 
Appendix  E  using  Microsoft  Excel  notation.  For  clarity,  those  predictor  variables  which 
are  only  applicable  to  historical  aircraft  with  contract  award  dates  prior  to  1985  are 
removed. 

Limitations 

We  recognize  several  major  limitations  that  could  potentially  limit  the 
applicability  of  this  research  to  “real  world”  cost  analysis  applications.  First,  continuing 
a  methodology  previously  applied  by  Brown  et  al.  (2002),  Porter  and  Gallagher  (2004), 
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and  Unger  et  al.  (2004),  we  utilize  the  published  OSD-Comptroller  outlay  rates  to 
estimate  expenditures  from  budgeted  TOA.  Therefore,  we  recognize  that  the  estimated 
program  expenditures  to  which  we  fit  our  models  are  not  “actuals”,  but  instead 
approximations  of  expenditures.  For  this  reason,  we  elect  to  apply  the  Lee  et  al.  (1997) 
methodology  to  transform  our  predicted  expenditures  into  predicted  TOA  before 
measuring  R  .  Similarly,  when  applying  the  models  presented  in  this  thesis  to  “real 
world”  estimates,  we  recommend  that  the  cost  analyst  also  apply  the  Lee  et  al. 
methodology  to  convert  their  predicted  expenditures  into  predicted  TOA. 

Second,  in  constructing  our  model,  we  utilize  budget  and  schedule  data  from  the 
latest  SAR  available  for  each  aircraft  development  program.  We  do  not  account  for  any 
cost  or  schedule  growth  which  exists  between  the  aircraft  program’s  first  and  latest  SAR. 
Therefore,  the  reported  Weibull  time  phasing  model  R“  of  0.746  assumes  absolutely 
accurate  inputs  for  the  cost,  schedule,  and  program  characteristics.  Once  again,  this 
assumption  contrasts  with  the  “real  world”,  where  we  recognize  that  cost  and  schedule 
estimates  are  rarely  clairvoyant.  As  a  result,  if  any  of  estimated  model  inputs  are  less 
than  accurate,  the  model  accuracy  will  decrease,  on  average,  below  0.746. 

Third,  we  recommend  that  the  cost  analyst  should  observe  caution  when  using  our 
models  to  “extrapolate”,  or  estimate  beyond  the  original  range  of  historical  observations. 
Due  to  the  usage  of  an  indicator  variable  which  distinguishes  between  pre-1985  contract 
awards  and  more  contemporary  programs,  we  further  restrict  this  recommendation  by 
asserting  that  the  cost  analyst  should  avoid  going  beyond  the  range  of  inputs  for 
contemporary  aircraft  given  in  Table  23,  particularly  for  the  Beta  model.  As  a 
demonstration  of  the  danger  of  interpolation,  the  reader  should  recognize  that  our  data  set 
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includes  only  four  new  starts  which  began  development  during  1985  or  later.  For  each  of 
these  new  starts,  the  length  of  development  was  between  9.3  and  14.8  years.  If  the  Beta 
model  is  extrapolated  against  a  theoretical  new  start  program  with  a  first  flight  at  40 
percent  schedule  and  a  comparatively  short  6  year  development  length,  we  discover  that 
it  estimates  only  20  percent  expenditures  at  50  percent  schedule,  which  is  unlikely. 
However,  it  should  be  distinguished  that  we  observe  that  our  Weibull  and  Rayleigh 
models  are  generally  more  stable  (compared  to  the  Beta)  when  extrapolated  outside  of  the 
observed  range. 


Table  23:  Upper  and  Lower  Bounds  for  Model  Inputs 


New  Start 

Upgrade 

All  Historic 

Contemporary 

All  Historic 

Contemporary 

Lower 

Upper 

Lower 

Upper 

Lower 

Upper 

Lower 

Upper 

First  Flight  (%  Schedule) 

0.29 

0.72 

0.34 

0.42 

0.28 

0.66 

0.39 

0.66 

Length  of  Development 

4.2 

14.8 

9.3 

14.8 

5.7 

12.2 

5.7 

12.2 

Recommendations  for  Future  Research 

We  recommend  two  areas  for  future  research.  First,  as  addressed  within 
limitations,  our  thesis  assumes  accurate  cost  and  schedule  inputs  without  growth. 
Therefore,  for  future  research,  we  recommend  the  creation  of  a  time  phasing  model  with 
the  capability  to  account  for  expected  cost  and  schedule  growth.  Secondly,  we  assert  that 
it  would  be  valuable  for  future  researchers  to  expand  upon  the  time  phasing  investigation 
completed  by  Unger  et  al.  (2004).  Unger  et  al.  find  that  a  statistically  significant 
relationship  exists  between  the  shape  of  the  initial  budget  profile  for  a  development 
program  and  observed  cost  and  schedule  growth.  However,  as  a  potential  limitation  of 
their  study,  Unger  et  al.  do  not  account  for  the  effect  of  budget  curtailments,  specifically 
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budget  “cuts”  within  the  early  years  of  development.  We  therefore  hypothesize  that  both 
1)  the  initial  shape  of  the  budget  (as  measured  by  Weibull  parameters)  and  2)  whether 
that  initial  budget  is  fully  funded  are  predictive  of  future  cost  and  schedule  growth. 

Summary 

Past  research  documents  that  funding  provided  in  the  wrong  fiscal  years  of  a 
development  program  result  in  productivity  inefficiencies,  schedule  slips  and  increased 
program  costs  (Belcher  et.  al,  1999;  Unger,  2001).  Therefore,  it  is  in  the  best  interest  of 
the  system  program  office  to  submit  an  appropriate  time  phased  estimate  to  the  budget 
formulation  process.  To  assist  in  the  estimation  process,  our  research  first  tests  the 
applicability  of  the  60/40  “rule  of  thumb”  to  contemporary  aircraft  programs.  We 
discover  that,  as  a  heuristic,  the  60/40  does  not  account  for  the  difference  is  spending 
patterns  between  new  start  and  upgrade  development  programs.  Contemporary  new  start 
programs  expend  a  mean  of  55  percent  expenditures  at  50  percent  schedule  (55/45),  while 
contemporary  upgrade  programs  expend  a  mean  of  65  percent  expenditures  at  50  percent 
schedule  (65/35).  Next,  we  construct  a  methodology  for  estimating  distribution 
parameters  for  the  Rayleigh,  Weibull,  and  Beta  using  characteristics  common  to  all 
aircraft  developing  programs.  We  find  that  time  to  first  flight,  years  of  development,  and 
upgrade  are  all  statistically  significant  predictors  of  our  distribution  parameters.  Finally, 
using  our  proposed  methodology,  we  identify  three  final  models  which  are  estimated  to 
improve  time  phasing  accuracy  when  compared  to  the  60/40  “rule  of  thumb”. 
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Appendix  A:  Weida  (1977)  Expenditure  Inflection  Points 


%  Expenditures 

%  Schedule 

B-l 

0.56 

0.36 

F-105 

0.58 

0.34 

C-141 

0.54 

0.51 

C-5 

0.52 

0.34 

XB-70 

0.47 

0.44 

Tug  R 

0.53 

0.55 

Tug  C 

0.50 

0.46 

TugE 

0.64 

0.51 

TugG 

0.66 

0.51 

C-5  Quality  Assurance  Hours 

0.44 

0.46 

C-5  Production  Hours 

0.52 

0.48 

A- 10  System 

0.49 

0.47 

A- 1 0  Engine 

0.69 

0.62 

A- 10  Gun 

0.56 

0.45 

A- 1 0  Milestones 

0.54 

0.42 

AGM-65  A 

0.62 

0.43 

AWACS 

0.70 

0.51 

Mean: 

0.56 

0.46 

Median: 

0.54 

0.46 

Std.  Deviation: 

0.08 

0.07 
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Appendix  B:  Estimated  Percent  Expenditures  at  50%  Schedule,  by  Program 


Program 

Y ear  of  EMD 

Percent  Expenditures  at  50%  Schedule 

Contract  Award 

Linear  Interpolation 

Weibull 

Beta 

A-10 

1973 

0.67 

0.68 

0.66 

AWACS  BLOCK  40 

2003 

0.69 

0.71 

0.69 

AWACS  RSIP 

1988 

0.80 

0.82 

0.82 

B-l  CMUP 

1996 

0.49 

0.51 

0.50 

B-2  EHF  1 

2007 

0.75 

0.78 

0.76 

B-2  RMP 

2004 

0.60 

0.62 

0.59 

B-2  A 

1981 

0.46 

0.46 

0.45 

C-17A 

1982 

0.36 

0.35 

0.35 

C-5  AMP 

1999 

0.57 

0.59 

0.58 

C-5  RERP 

2001 

0.76 

0.77 

0.76 

E-2D 

2004 

0.70 

0.71 

0.69 

E-6A 

1983 

0.51 

0.54 

0.52 

EA-18G 

2003 

0.49 

0.50 

0.48 

EA-6B  ICAP 

1998 

0.56 

0.56 

0.54 

F-14A 

1969 

0.55 

0.57 

0.54 

F-14D 

1984 

0.51 

0.50 

0.49 

F-15A 

1970 

0.47 

0.49 

0.47 

F-16A/B 

1975 

0.55 

0.55 

0.54 

F-18E/F 

1992 

0.77 

0.77 

0.76 

F-18A 

1976 

0.38 

0.38 

0.38 

F-22 

1991 

0.60 

0.63 

0.60 

F-35  AF 

2001 

0.60 

0.61 

0.58 

F-5E 

1970 

0.37 

0.35 

0.35 

J  STARS 

1985 

0.54 

0.59 

0.56 

P-8A 

2004 

0.46 

0.47 

0.46 

T-46A 

1982 

0.43 

0.45 

0.43 

Mean: 

0.56 

0.58 

0.56 

Median: 

0.55 

0.57 

0.54 

Std.  Deviation: 

0.13 

0.13 

0.13 
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Appendix  C:  Estimated  Distribution  Parameters  by  Program 


Rayleigh  - 

Weibull  - 

Weibull  - 

Beta  - 

Beta  - 

Program 

Scale  (3 

Shape  a 

Scale  P 

Shape  a 

A-10 

0.46 

1.65 

0.46 

1.24 

1.84 

AWACS  Block  40 

0.44 

1.45 

0.43 

1.09 

1.80 

AWACS  RSIP 

0.37 

1.76 

0.37 

1.78 

3.73 

B-l  CMUP 

0.57 

2.46 

0.57 

2.21 

2.20 

B-2  EHF  1 

0.39 

1.52 

0.38 

1.25 

2.46 

B-2  RMP 

0.51 

1.87 

0.51 

1.47 

1.85 

B-2  A 

0.60 

2.54 

0.60 

2.25 

2.01 

C-17A 

0.64 

3.67 

0.63 

4.47 

3.46 

C-5  AMP 

0.52 

2.39 

0.52 

2.30 

2.71 

C-5  RERP 

0.41 

1.86 

0.41 

1.84 

3.30 

E-2D 

0.44 

1.71 

0.44 

1.46 

2.33 

E-6A 

0.56 

2.19 

0.56 

1.76 

1.83 

EA-18G 

0.59 

2.18 

0.59 

1.67 

1.57 

EA-6B  1CAP 

0.55 

2.10 

0.55 

1.81 

1.99 

F-14A 

0.55 

1.99 

0.55 

1.53 

1.69 

F-14D 

0.58 

2.63 

0.58 

2.66 

2.61 

F-15A 

0.59 

2.42 

0.59 

1.98 

1.86 

F-16A/B 

0.54 

2.40 

0.55 

2.31 

2.51 

F-18A 

0.64 

3.10 

0.64 

3.09 

2.41 

F-18E/F 

0.40 

1.76 

0.40 

1.71 

3.13 

F-22 

0.51 

1.75 

0.50 

1.32 

1.70 

F-35  AF 

0.52 

1.87 

0.52 

1.51 

1.86 

F-5E 

0.68 

2.59 

0.69 

1.31 

0.85 

JSTARS 

0.54 

1.68 

0.54 

1.14 

1.33 

P-8A 

0.60 

2.29 

0.61 

1.72 

1.55 

T-46A 

0.61 

2.63 

0.61 

2.24 

1.93 

Mean: 

0.53 

2.17 

0.53 

1.86 

2.13 

Median: 

0.55 

2.10 

0.55 

1.72 

1.93 

Std.  Deviation: 

0.08 

0.50 

0.09 

0.72 

0.70 
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Appendix  D:  Tests  for  Influence,  Normality,  and  Constant  Variance 


Rayleigh  Scale  Parameter  (3  -  Influential  Data  Points 


Rayleigh  Scale  Parameter  P-  Normality  of  Residuals 


6  i 


Shapiro-Wilk  W  Test 
Prob.  <W:  0.9497 
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Rayleigh  Scale  Parameter  P  -  Constant  Variance 


0.4  0.45  0.5  0.55  0.6  0.65  0.7 

Predicted 


Breusch -Pagan  Test: 

P-value  -  0.5264 

Weibull  Shape  Parameter  a  -  Influential  Data  Points 
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Weibull  Shape  Parameter  a  -  Normality  of  Residuals 


Shapiro-Wilk  W  Test 
Prob.  <  W:  0.9849 


Weibull  Shape  Parameter  a  -  Constant  Variance 


Breusch -Pagan  Test: 

P-value  -  0.691 1 
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Weibull  Scale  Parameter  P  -  Influential  Data  Points 


Weibull  Scale  Parameter  P  -  Normality  of  Residuals 
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Weibull  Scale  Parameter  P  -  Constant  Variance 


Breusch -Pagan  Test: 

P-value  -  0.5176 


Beta  Shape  Parameter  a  -  Influential  Data  Points 
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Beta  Shape  Parameter  a  -  Normality  of  Residuals 


-1  -0.75  -0.5  -0.25  0  0.25  0.5  0.75 

Shapiro-Wilk  W  Test 
Prob.  <  W:  0.4399 


Beta  Shape  Parameter  a  -  Constant  Variance 


Breusch-Pagan  Test: 

P-value  -  0.6473 
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Beta  Shape  Parameter  P  -  Influential  Data  Points 


Beta  Shape  Parameter  P  -  Normality  of  Residuals 
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Beta  Shape  Parameter  P  -  Constant  Variance 


Breusch -Pagan  Test: 

P-value  -  0.4188 
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Appendix  E:  Functional  Forms  of  Final  Models  (Excel  Notation) 

Note:  For  easier  application  to  contemporary  aircraft,  input  variables  which  are 
applicable  only  to  aircraft  with  pre-1985  contract  awards  have  been  removed  from 
models. 

Final  Rayleigh  Distribution  Model 

—  Weibull(%  schedule,  a,  /?,  true ) 
a  —  2 

/?  =  0.45026  +  0.2404  (%  Schedule  at  First  Flight ) 

—  0.07819  (0  if  New  Start ;  1  if  Upgrade ) 

Final  Weibull  Distribution  Model 

—  Weibull(%  schedule,  a,  ft,  true ) 
a  =  1.91047 

/?  =  0.44265  +  0.2587  (%  Schedule  at  First  Flight ) 

—  0.07906  (0  if  New  Start)  1  if  Upgrade ) 

Final  Beta  Distribution  Model 

—  Betadist(%  schedule,  a,  /?) 

a  =  -  1.80324  -  0.32456  ( Length )  -  2.94584  (In  ) 

LiGFIQ  t  h 

P  =  -  0.36129  +  0.155517  ( Length ) 

+  1.49292  (0  if  New  Start ;  1  if  Upgrade ) 
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14.  ABSTRACT 

Early  research  on  time  phasing  primarily  focuses  on  the  theoretical  foundation  for  applying  the  continuous 
distribution  function,  or  S-curve,  to  model  the  distribution  of  development  expenditures.  Minimal 
methodology  is  provided  for  estimating  the  S-curve’s  parameter  values.  Brown,  White,  and  Gallagher 
(2002)  resolve  this  shortcoming  through  regression  analysis,  but  their  methodology  has  not  been  widely 
adopted  by  aircraft  cost  analysts,  as  it  is  judged  as  overly  broad  and  not  specific  to  aircraft.  Instead, 
analysts  commonly  apply  the  60/40  “rule  of  thumb”  to  aircraft  development,  assuming  60  percent 
expenditures  at  50  percent  schedule.  It  is  currently  unknown  if  the  60/40  heuristic  accurately  describes 
contemporary  aircraft  development  programs.  Therefore,  using  a  sample  of  26  DoD  aircraft  programs,  we 
first  test  the  accuracy  of  60/40,  discovering  that,  as  a  heuristic,  the  60/40  cannot  account  for  differences 
between  new  start  and  upgrade  programs.  Next,  we  improve  upon  prior  research  by  using  program 
characteristics  to  construct  an  aircraft-specific  methodology  for  estimating  parameters.  Finally,  we 
conclude  our  research  by  comparing  the  accuracy  of  our  Rayleigh,  Weibull,  and  Beta  S-curve  models.  Our 
Weibull  model  explains  74.6  percent  of  total  variation  in  annual  budget,  improving  the  estimation  of  budgets 
by  6.5  percent,  on  average,  over  the  baseline  60/40  model. _ 
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